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How  to  Use  This  Guide 


This  guide  provides  you  with  three  tools  to  develop  performance  measures  for  your  project, 
program,  or  acquisition: 

1.  detailed  procedures  on  how  to  develop  performance  measures; 

2.  examples  of  methodologies  and  performance  measures;  and 

3.  a  case  study  demonstrating  how  performance  measures  are  implemented  in  a  hypothetical 
automated  information  system  (AIS)  development  project. 

These  tools,  in  combination  with  your  technical,  managerial,  and  functional  area  experience,  will 
help  you  develop  a  performance  measurement  program.  The  guide  is  organized  into  seven 
sections  and  a  set  of  appendices. 

Section  1  introduces  the  legislative  framework  and  management  rationale  for  performance 
measurement. 

Section  2  introduces  the  IT  investment  process 

Section  3  provides  an  overview  of  performance  measures  and  identifies  the  major  audiences  for 
IT  performance  measures. 

Sections  4-6  describe  procedures  for  development  of  performance  measures  at  the  enterprise, 
program,  and  program/project  levels  which  will  be  integrated  into  the  Selection,  Control  and 
Evaluation  of  the  IT  investment  process  described  in  Section  2.  Even  if  an  effort  is  well 
established,  it  is  not  too  late  to  begin  an  active  performance  measures  program.  Follow  the  steps 
in  this  section  that  apply.  Use  the  example  measures  as  a  guide  to  think  about  benefits,  schedule, 
and  cost.  Remember,  this  section  is  a  guide,  not  a  set  of  mandatory  steps. 

Section  7  contains  a  case  study  providing  examples  of  how  each  step  described  in  Sections  4-6 
has  been  applied  in  a  hypothetical  IM/IT  systems  project. 

Section  8  discusses  some  proven  methodologies  for  measuring  IT  performance. 

Appendix  A  is  the  proposed  Investment  Baseline/Performance  Agreement 

The  bibliography  in  Appendix  B  contains  a  listing  of  reference  materials.  Use  this  list  to  select 
additional  readings  in  areas  where  you  need  more  guidance. 

Most  important ...  get  going.  Start  now  wherever  you  are.  By  using  the  information 
contained  in  this  guide,  you  will  be  able  to  make  the  theory  real  and  put  performance 
measurement  into  practice. 
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Executive  Summary 


This  guide  summarizes  the  Department  of  Defense  (DoD)  position  on  Information  Technology 
(IT)  performance  measurement  and  presents  a  framework  for  managing  information  technology 
programs  as  investments  rather  than  as  acquisitions. 

DoD’s  goal  is  to,  within  the  framework  of  Government  Performance  and  Results  Act  (GPRA), 
the  Information  Technology  Management  and  Results  Act  (ITMRA)  and  other  relevant 
management  legislation,  establish  performance  measures  as  an  integral  part  of  the  Information 
Technology  (IT)  investment  process.  It  is  designed  to  assist  DoD  in  its  transition  to  a 
performance-based  organization. 

Managing  IT  as  an  investment  requires  that  senior  managers  be  able  to  systematically  maximize 
the  benefits  of  IT  investments  throughout  the  organization  by  using  the  three  steps  of  the  IT 
investment  process: 

•  Selection  -  Creating  a  portfolio  of  IT  project  investments  that  maximizes  mission 
performance,  using  an  approved  set  of  criteria  for  consistent  comparison  of  projects. 

•  Control  -  Measuring  ongoing  IT  projects  against  their  projected  costs,  schedules,  and 
benefits  and  taking  action  to  continue,  modify,  or  cancel  them. 

•  Evaluation  -  Determining  the  actual  value  of  an  implemented  investment  against  the 
organization’s  mission  requirements  and  adapting  the  IT  investment  process  to  reflect 
lessons  learned. 

Essential  to  the  successful  use  of  this  process  is  an  effective  means  of  measuring  the  performance 
of  IT  investments  in  objective,  outcome-oriented  terms.  The  guide  defines  IT  performance 
measurement  as: 

the  assessment  of  effectiveness  and  efficiency  of  IT  in  support  of  the  achievement 
of  an  organization ’s  missions,  goals,  and  quantitative  objectives  through  the 
application  of  outcome-based,  measurable,  and  quantifiable  criteria,  compared 
against  an  established  baseline,  to  activities,  operations,  and  processes. 

Performance  Measurement  is  the  means  by  which  an  organization  measures  its  effectiveness  and 
efficiency  in  the  pursuit  of  its  missions,  goals,  and  objectives. 

The  six  key  steps  in  effective  performance  measurement  are: 

1.  Define  mission,  key  result  areas,  and  business  functions. 

2.  Develop  mission  related  goals. 

3.  Generate  performance  measures/indicators. 

4.  Validate  and  verify  performance  measures. 

5.  Implement  the  performance  measures  and  collect  data. 

6.  Monitor  and  assess  the  results  and  repeat  the  process  as  needed. 
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There  is  a  recognition  that  different  management  tiers  need  different  kinds  of  information  to 
make  investment  and  business  decisions. 

Enterprise  Level:  At  the  Enterprise  level ,  mission  results  are  being  focused  on,  and  information 
is  needed  to  choose  policy  directions  and  make  mission  decisions.  Managers  make  the 
connection  between  the  IT  performance  measurement  requirements  of  ITMRA  and  the  overall 
organizational  performance  measurement  required  by  GPRA.  IT  investment  decisions  are 
greatly  influenced  by  this  level  during  the  Selection  phase  of  the  investment  process.  The  timing 
for  information  is  cyclical. 

Relationship:  Policy,  mission  decision  and  strategies 

Role:  Accountability 

Timing:  Cyclical 

Functional  level:  At  the  Functional  level,  the  focus  is  on  unit  results  where  information  is 
needed  to  manage  and  improve  operations.  These  managers  are  responsible  for  reporting  the 
performance  of  major  DoD  functions  across  multiple  projects,  programs,  or  acquisitions.  They 
combine,  synthesize  and  report  program/project-level  results  for  use  by  enterprise-level 
managers.  The  functional  level  is  also  where  mission  related  outcome  measures  are  defined  and 
the  interests  of  the  IM/IT  user  community  are  directly  represented,  and  their  requirements  are 
approved,  documented,  and  funded  for  execution.  This  level  is  heavily  involved  in  the  Selection 
and  Evaluation  phases  of  the  investment  process.  The  program/project  level  serves  as  the 
Functional  Level  representative  during  the  Control  phase  of  the  investment  process.  The  timing 
for  information  is  periodic. 

Relationship:  Management  and  improvement  of  operations 
Role:  Integration  &  Planning 

Timing:  Periodic 

Prograin/project  level :  At  the  Program/project  level,  activity  and  task  information  is  critical  to 
make  tactical  decisions  and  execute  management  decisions.  Managers  are  leading  programs, 
projects,  or  acquisitions  that  are  sponsored  by  functional-level  managers.  These  managers  are 
involved  in  the  accomplishment  of  actual  IM/IT  efforts.  This  level  serves  a  key  role  in  the 
Control  and  Evaluation  phases  of  the  investment  process.  The  timing  for  information  is 
immediate. 

Relationship:  Tactical  and  execution  management 
Role:  Resource  allocation 

Timing:  Immediate 

For  each  of  these  levels,  the  guide  lays  out  a  process  for  defining  IT  investments  in  terms  of 
functional  requirements,  identifying  outcome-based  performance  measures  that  accurately  assess 
the  achievement  of  these  requirements,  and  continuously  ensuring  that  the  linkage  between 
investments  and  mission  accomplishment  is  maintained.  The  functional  level  in  concert  with  the 
users,  are  responsible  for  conducting  post-implementation  assessment  of  IT  investments’ 
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operational  capabilities,  to  ensure  that  the  expected  functional  benefits  of  an  investment  are 
actually  realized.  This  post-deployment  assessment  is  crucial  to  ensuring  that  the  linkage 
between  IT  investments  and  organizational  performance  is  maintained. 

The  process  culminates  in  the  development  of  an  Investment  Baseline/Performance  Agreement 
by  all  levels  of  management  involved  in  an  IT  investment.  The  Investment  Baseline/Performance 
Agreements’  performance  parameters  represent  the  minimum  number  needed  to  characterize  the 
major  drivers  of  operational  effectiveness,  suitability,  schedule,  technical  progress  and  cost.  This 
minimum  number  include  the  key  outcome  measures  described  in  the  requirements  definition 
document. 

The  Investment  Baseline  Performance  Agreement  captures  the  functional  performance,  cost  and 
schedule  baselines  for  the  project,  the  outcome  measures  that  will  be  used  to  evaluate  the 
project’s  success,  and  the  linkage  back  to  the  organization’s  strategic  goals.  Jointly  validated  by 
the  CIO  and  CFO  (or  their  equivalents),  the  user  representative/functional  manager,  and  the 
project  manager,  The  Investment  Baseline/Performance  Agreement  firmly  ties  each  IT 
investment  to  a  set  of  clearly  understood,  quantifiable,  outcome-based  performance  measures  that 
directly  support  mission  accomplishment. 

The  methods  and  measures  described  in  this  guide  are  widely  used  for  program  management 
purposes.  However,  they  are  not  meant  to  imply  an  exhaustive  or  required  set  of  measures  or 
methods.  The  guide  does  lay  out  basic  principles  and  processes  that  users  can  tailor  to  their  set 
of  circumstances. 
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1.  INTRODUCTION 


This  document  summarizes  the  Department  of  Defense  (DoD)  position  on  Information 
Technology  (IT)  performance  measurement  and  presents  a  framework  for  IT  program  managers 
to  manage  their  IT  programs  as  investments.  Section  1  presents  the  purpose  and  legislative 
background  of  this  document. 

1.1  Purpose 

As  the  Department  of  Defense  (DoD)  becomes  a  performance-based  organization  (PBO), 
demonstrating  results  becomes  the  basis  for  success,  not  simply  spending  allocated  budgets. 
Increased  public  scrutiny,  tighter  budgets,  and  legislative  mandates  all  compel  Information 
Technology  (IT)  managers  to  focus  their  attention  on  managing  IT  investments,  rather  than 
focusing  too  narrowly  on  IT  acquisitions.  The  emphasis  must  be  on  achieving  outcomes  that 
contribute  to  mission  effectiveness,  rather  than  simply  meeting  contractual  requirements.  To 
demonstrate  success,  each  program,  project,  and  acquisition  must  institutionalize  outcome- 
oriented  performance  measures;  performance  must  be  evaluated  over  time  using  these  measures. 

This  guide  sets  out  an  analytical  framework  for  linking  IT  investment  decisions  to  strategic 
objectives,  business  plans,  and  organizational  mission  performance,  While  this  guide  supports 
DoD  methods  and  processes,  it  does  not  depend  on  the  adoption  of  any  specific  standard.  This 
guide  provides  a  flexible  framework  for  integrating  measurements  into  existing  management  and 
development  processes.  We  strongly  encourage  its  use  to  identify  strengths  and  weaknesses  of 
current  IT  investment  control  systems  and  to  develop  action  plans  for  improvement. 

1.2  Legislative  Background 

Congress  has  determined  that  waste  and  inefficiency  in  Federal  Information  Technology  (IT) 
programs  undermine  the  confidence  of  the  American  people  and  reduces  the  Federal 
Government’s  ability  to  adequately  address  vital  public  needs.  Federal  IT  managers  are  seriously 
disadvantaged  in  their  efforts  to  improve  IT  program  efficiency  and  effectiveness,  because  of 
insufficient  articulation  of  program  goals  and  inadequate  information  on  program  performance. 
Furthermore,  Congressional  policymaking,  spending  decisions,  and  program  oversight  are 
seriously  handicapped  by  insufficient  attention  to  program  performance  and  results.  Congress 
recognizes  existing  planning  and  management  systems  have  not  accomplished  their  original 
missions  for  IT  programs.  Critical  functions  such  as  planning,  budgeting,  program 
implementation  and  program  evaluation  are  disjointed.  For  many  IT  programs  any  linkage 
between  agency-wide  goals,  budgets,  implementation  activities,  and  performance  measures  is 
purely  coincidental. 

To  correct  the  shortcomings  associated  with  current  management  of  IT  investments,  Congress 
enacted  several  pieces  of  legislation  requiring  Federal  Agencies  to  implement  performance 
measures  in  their  business  processes  to  ensure  the  proper  oversight  and  management  of  IT 
investments.  Performance  measures  are  now  required  by  law  to  be  an  integral  part  of  any  IT 
program  This  legislation  has  been  followed  by  Executive  Orders  and  OMB  Circulars  regarding 
performance  measurement.  These  documents  are  summarized  below. 
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1.2.1  Information  Technology  Management  Reform  Act  of  1996 

As  part  of  the  National  Defense  Authorization  Act  For  Fiscal  Year  1996,  the  Information 
Technology  Management  Reform  Act  of  1996  (ITMRA)  mandates  that  the  Secretary  of  Defense 
implement  performance  measures  for  all  DoD  information  technology  (IT)  programs,  projects, 
and  acquisitions.  This  requirement  also  applies  to  National  Security  Systems. 

This  act  requires  the  Secretary  of  Defense  to  design  and  implement  a  process  for  maximizing  the 
value  and  assessing  and  managing  the  risks  of  the  FT  investments  of  the  DoD.  The  Secretary  of 
Defense  is  required  to  provide  the  means  for  senior  external  management  personnel  to  obtain 
timely  information  regarding  investment  progress  in  an  information  system.  This  includes  a 
system  of  milestones  for  measuring  progress  on  an  independently  verifiable  basis  in  terms  of 
cost,  timeliness,  quality,  and  system  capabilities  versus  requirements. 

The  Chief  Information  Officer  (CIO)  of  DoD  must  monitor  the  performance  of  IT  programs  of 
the  agency,  evaluate  the  performance  of  those  programs  on  the  basis  of  the  applicable 
performance  measurements,  and  advise  the  head  of  the  agency  regarding  whether  to  continue, 
modify,  or  terminate  a  program  or  project. 

The  Secretary  of  Defense  will  report  the  program  performance  benefits  resulting  from  FT  capital 
investments  to  the  Director  of  the  Office  of  Management  and  Budget  (OMB).  The  Secretary  of 
Defense  will  also  report  how  these  benefits  relate  to  the  accomplishment  of  DoD’s  goals.  OMB 
will  compare  DoD’s  performance  to  the  performance  of  other  executive  agencies. 

The  act  requires  the  DoD  to  use  performance  and  results-based  management.  One  method  that 
OMB  will  use  to  implement  this  requirement  is  periodic  reviews  of  selected  IM/IT  activities  as 
part  of  the  budget  process.  OMB  will  enforce  accountability  through  the  budgetary  process. 

With  respect  to  National  Security  Systems,  ITMRA  does  not  require  DoD  to  fully  implement 
OMB  direction  on  FT  management.  Rather,  the  Secretary  of  Defense  has  the  authority  to  apply 
this  direction  to  the  extent  practicable,  while  accomplishing  vital  military  and  intelligence 
missions.  ITMRA  expressly  provides  this  authority  to  be  used  at  the  Secretary’s  discretion. 

1.2.2  Section  381  of  the  National  Defense  Authorization  Act  For  Fiscal  Year  1995 

Section  381  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  1995  requires  DoD  to 
establish  performance  measures  and  management  controls  to  supervise  and  manage  its  IM/IT 
activities.  Specifically: 

“(1)  The  Secretary  of  Defense  shall  establish  performance  measures  and 
management  controls  for  the  supervision  and  management  of  the  activities  ...  The 
performance  measures  and  management  controls  shall  be  adequate  to  ensure,  to 
the  maximum  extent  practicable,  that  the  Department  of  Defense  receives  the 
maximum  benefit  possible  from  the  development,  modernization,  operation,  and 
maintenance  of  automated  information  systems.” 
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The  act  also  requires  the  Secretary  of  Defense  to  report  on  the  establishment  and  implementation 
of  the  performance  measures  and  management  controls  in  1995,  1996,  and  1997. 

1.2.3  Federal  Acquisition  Streamlining  Act  (FASA)  of  1994 

Title  V  of  the  FASA  contains  specific  requirements  for  federal  agencies  to  “define  the  cost 
performance  and  schedule  goals  for  major  acquisition  programs”  and  to  monitor  and  report 
annually  on  the  degree  to  which  these  goals  are  being  met.  In  their  annual  reports,  agencies  must 
assess  whether  acquisition  programs  are  achieving  90%  of  their  cost,  performance,  and  schedule 
goals.  If  not,  agencies  must  determine  whether  these  programs  should  continue.  The  FASA  also 
provides  for  an  enhanced  system  of  performance  incentives  to  relate  performance  to  the 
achievement  of  these  goals. 

1.2.4  Government  Performance  and  Results  Act  (GPRA)  of  1993 

This  legislation  requires  strategic  planning  and  performance  measurement  in  the  executive 
branch  agencies  of  the  federal  government.  The  purposes  of  the  GPRA  are  to  improve  federal 
management  and  congressional  decision  making,  service  delivery,  program  effectiveness  and 
public  accountability,  and  public  confidence  in  government.  The  GPRA  requires  agencies  to 
develop  strategic  plans  by  September  30,  1997,  for  implementation  in  fiscal  year  1999.  The 
OMB  has  mandated  that  the  plans  cover  six  years  and  be  updated  at  least  every  three  years. 
Stakeholders  and  customers  will  provide  input  into  the  strategic  plans. 

Beginning  in  fiscal  year  1999,  agencies  will  develop  yearly  performance  plans  and  set 
performance  goals  based  on  their  strategic  plans.  Starting  in  March  2000,  agencies  will  write 
annual  performance  reports,  comparing  actual  performance  to  goals  established  in  annual 
performance  plans. 

This  performance  information  gives  IM/IT  managers  two  advantages:  (1)  an  early  warning 
system  to  improve  program  management  and  (2)  information  communicating  program  value  to 
executives,  Congress,  other  stakeholders,  and  the  general  public. 

1.2.5  Paperwork  Reduction  Act  (PRA)  of  1995 

This  legislation  is  intended  to  minimize  the  paperwork  burden  resulting  from  the  collection  of 
information  by  or  for  the  Federal  Government;  coordinate,  integrate,  and  make  uniform  Federal 
information  resources  management  policies  and  practices;  improve  the  quality  and  use  of 
Federal  information,  to  minimize  the  cost  to  the  Federal  Government  of  the  creation,  collection, 
maintenance,  use,  dissemination,  and  disposition  of  information,  and  to  ensure  that  information 
technology  is  acquired,  used,  and  managed  to  improve  performance  of  Federal  agency  missions. 

The  Act  requires  that  each  agency: 

•  Develop  and  maintain  a  strategic  information  resources  management  plan  that  shall  describe 
how  information  resources  management  activities  help  accomplish  agency  missions 

•  Develop  and  maintain  an  ongoing  process  to— 
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1.  ensure  that  information  resources  management  operations  and  decisions  are  integrated 
with  organizational  planning,  budget,  financial  management,  human  resources 
management,  and  program  decisions; 

2.  in  cooperation  with  the  agency  Chief  Financial  Officer  (or  comparable  official),  develop  a 
full  and  accurate  accounting  of  information  technology  expenditures,  related  expenses, 
and  results. 

3.  establish  goals  for  improving  information  resources  management's  contribution  to 
program  productivity,  efficiency,  and  effectiveness,  methods  for  measuring  progress 
towards  those  goals,  and  clear  roles  and  responsibilities  for  achieving  those  goals. 

4.  Maintain  a  current  and  complete  inventory  of  the  agency's  information  resources. 

5.  Conduct  formal  training  programs  to  educate  agency  program  and  management  officials 
about  information  resources  management. 

These  provisions  of  the  PRA  were  reinforced  and  expanded  by  the  ITMRA  in  1996. 


1.2.6  Chief  Financial  Officers’  Act  (CFO A)  of  1990 

This  legislation  was  enacted  to  accomplish  these  objectives: 

1.  Bring  more  effective  general  and  financial  management  practices  to  the  Federal 
Government  through  statutory  provisions  which  would  establish  in  the  Office  of 
Management  and  Budget  a  Deputy  Director  for  Management,  establish  an  Office  of 
Federal  Financial  Management  headed  by  a  Comptroller,  and  designate  a  Chief  Financial 
Officer  in  each  executive  department  and  in  each  major  executive  agency  in  the  Federal 
Government. 

2.  Provide  for  improvement,  in  each  agency  of  the  Federal  Government,  of  systems  of 
accounting,  financial  management,  and  internal  controls  to  assure  the  issuance  of  reliable 
financial  information  and  to  deter  fraud,  waste,  and  abuse  of  Government  resources. 

3.  Provide  for  the  production  of  complete,  reliable,  timely,  and  consistent  financial 
information  for  use  by  the  executive  branch  of  the  Government  and  the  Congress  in  the 
financing,  management,  and  evaluation  of  Federal  programs. 

It  requires  agencies  to  include  performance  measurement  data  in  their  annual  financial 
statements. 

1.2.7  OMB  Circular  A-ll,  Part  2:  Preparation  and  Submission  Of  Strategic  Plans 

This  circular  provides  executive  guidance  for  preparing  and  submitting  Agency  strategic  and 
performance  plans  as  required  by  GPRA. 

1.2.8  OMB  Circular  A-ll,  Part  3:  Planning,  Budgeting,  and  Acquisition  of  Fixed  Assets 

This  circular  provides  executive  guidance  on  planning  ,  budgeting,  and  acquisition  of  fixed 
assets,  specifically  FT  and  NSS-IT,  in  accordance  with  GPRA  and  ITMRA.  It  requires  agencies 
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to  identify  baseline  goals  for  cost,  schedule,  and  performance  for  all  proposed  and  ongoing 
acquisitions,  and  provides  guidance  on  reporting  compliance  with  these  goals  to  OMB. 

1.2.9  OMB  Circular  A-130:  Management  of  Federal  Information  Resources 

This  circular  provides  executive  guidance  on  the  management  of  Federal  IM/IT  resources  in 
compliance  with  PRA  95.  Specific  requirements  include  strategic  IM/IT  planning  tying  IT 
investments  to  agency  mission  accomplishment  and  cost-benefit-analysis  of  IT  systems 
throughout  the  system  life-cycle. 

1.2.10  Executive  Order  13011,  Federal  Information  Technology 

This  order  implements  the  provisions  of  ITMRA  in  the  executive  branch.  Besides  the  specific 
provisions  of  ITMRA,  the  order  establishes  the  Federal  CIO  Council;  creates  the  Government 
Information  Technology  Services  Board  and  the  Information  Technology  Resources  Board;  and 
provides  additional  guidance  on  the  roles  of  agency  CIOs  and  the  use  performance  measurement 
in  evaluating  IT  investments. 
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2.  MANAGING  IT  AS  AN  INVESTMENT 


To  help  DoD  and  other  Federal  Agencies  achieve  their  goals,  the  General  Accounting  Office 
(GAO)  studied  successful  private  and  public  sector  organizations  to  learn  the  factors  behind  their 
success  in  results-based  management  of  IT  investments.  GAO  and  the  Office  of  Management 
and  Budget  (OMB)  created  an  IT  Investment  Guide  which  DoD  fully  endorses  as  a 
recommended  method  of  managing  IT  as  an  investment  within  the  Department.  The  key 
provisions  of  the  GAO/OMB  Investment  Guide  are  incorporated  into  this  section. 

2.1  The  IT  Investment  Management  Process 

This  section  describes  the  critical  success  elements  and  key  phases  that  should  be  a  part  of  a 
mature  IT  investment  process.  The  IT  investment  process  created  in  your  organization  should 
match  the  culture  and  organizational  structure.  The  overriding  objective  is  that  senior  managers 
be  able  to  systematically  maximize  the  benefits  of  IT  investments  throughout  the  organization 
and  establish  performance  measures  as  an  integral  part  of  the  IT  investment  process.  Details  are 
provided  below  to  describe  the  integration  of  performance  measures,  define  the  process  used  to 
link  DoD’s  Planning,  Programming  and  Budgeting  System  (PPBS)  to  the  Life  cycle  Management 
Process  and  the  departments  approach  to  Evaluating  the  Operational  Capability  of  IT  investments 
once  they’ve  been  deployed. 

The  process  starts  with  prioritizing  funding  requests  to  maximize  the  value  of  scarce  resources. 
This  difficult  process  involves  balancing  potential  benefits  against  costs  and  risks  and  aligning 
strategic  and  tactical  goals  with  proposed  system  investments.  Equally  critical,  the  approach 
ends  with  clear  evidence  of  positive  net  benefit  to  the  department  for  dollars  invested. 

As  described  below,  the  three  phases  of  the  investment  process  occur  in  a  continuous  cycle  of 
selection,  control  and  evaluation.  Information  from  each  phase  flows  freely  among  all  of  the 
other  phases  with  the  exception  of  evaluation.  The  evaluation  component  of  the  process  has  a 
unidirectional  information  flow  to  the  selection  component.  The  evaluation  component  is  used 
to  verify  or  modify  the  criteria  used  during  selection. 

1.  Investment  Selection  -  Creating  a  portfolio  of  IT  project  investments  that  maximizes 
mission  performance,  using  an  approved  set  of  criteria  for  consistent  comparison  of 
projects. 

2.  Investment  Control  -  Measuring  ongoing  IT  projects  against  their  projected  costs, 
schedules,  and  benefits  and  taking  action  to  continue,  modify,  or  cancel  them. 

3.  Investment  Evaluation  -  Determining  the  actual  value  of  an  implemented  investment 
against  the  organization’s  mission  requirements  and  adapting  the  IT  investment  process  to 
reflect  lessons  learned. 

The  control  and  evaluation  phases  are  conducted  throughout  the  year  and  their  results  are  fed  into 
the  selection  phase,  which  in  turn  feeds  back  to  the  control  and  evaluation  phases. 
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Exhibit  2-1:  IT  Investment  Management  Processes 


2.2  Organizational  Attributes  for  Successful  IT  Investments 

While  each  phase  of  the  IT  investment  management  process  has  unique  requirements  for 
successful  implementation,  there  are  some  overall  organizational  attributes  that  are  essential  for 
effective  investment  management.  These  shared  attributes  are:  senior  management  attention, 
overall  mission  focus,  and  a  comprehensive  portfolio  approach  to  IT  investment. 

2.2.1  Senior  Management  Attention 

Organizations  must  ensure  that  senior  managers,  with  the  authority  to  make  decisions  to 
continue,  modify,  or  cancel  IT  investment  programs  are  continuously  involved  in  the  IT 
investment  process. 


There  must  be  a  disciplined  decision-making  process  with  the  capability  to  approve,  cancel,  or 
delay  projects,  mitigate  risks,  and  validate  expected  returns  on  IT  investments. 


There  must  be  clearly  defined  roles,  responsibilities,  and  accountability  for  the  success  of  IT 
investments.  Formal  agreements  between  CIOs,  CFOs,  program  managers  and  users  of  IT 
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should  be  established;  IT  issues  and  requirements  must  be  integrated  into  financial  and 
operational  strategic  planning;  and  the  CFO  and  CIO  offices  must  be  involved  in  IT  operational 
decisions. 

2.2.2  Overall  Mission  Focus 

The  strategic  plan  is  the  organization’s  mission,  goals,  and  objectives,  and  performance 
measures  must  be  linked  to  strategic  planning  as  required  by  ITMRA  and  GPRA.  This  requires 
developing  long-term  strategic  goals,  setting  annual  organizational  performance  targets  in 
support  of  those  objectives,  and  annually  evaluating  performance  against  those  targets. 

GPRA  and  ITMRA  require  clear  organizational  hierarchies  of  goals  and  performance  measures 
be  established.  To  comply  with  the  spirit  and  intent  of  both  laws,  the  goals  and  measures  used  at 
lower  organizational  levels  must  be  linked  to  DoD’s  mission/strategic  goals.  Mission  goals 
should  be  translated  into  objective,  results-oriented  measures  of  performance  to  establish  a 
baseline  for  measuring  the  value  of  IT  investments.  Research  has  indicated  that  without  clear 
hierarchically  linked  goals  and  performance  measures,  managers  and  staff  throughout  the 
organization  lack  straightforward  roadmaps  showing  how  their  work  contributes  to  attaining 
organizational  strategic  goals. 

If  an  IT  investment  does  not  measurably  improve  agency  mission  performance  (  no  matter  how 
well  the  program  met  its  cost  and  schedule  baselines  or  output  and  performance  indicator 
measures),  that  investment  should  not  be  made. 

Agencies  must  ensure  that  functions  performed  by  proposed  IT  investments  are  mission-essential 
and  cannot  be  performed  more  efficiently  by  other  Government  or  private  activities.  The  work 
processes  supported  by  proposed  IT  investment  must  be  reviewed  and,  if  necessary,  reengineered 
to  ensure  the  full  value  of  the  investment  is  realized. 

Mission  benefit,  not  cost  and  schedule  constraints,  must  be  the  overriding  measure  of 
success  for  any  IT  project.  It  is  how  IT  contributes  to  mission  accomplishment  that  is  the 
deciding  factor  for  investment  purposes. 

2.2.3  Comprehensive  Portfolio  Approach  to  IT  Investment 

Organizations  must  define  their  portfolio  of  IT  investments  in  every  phase  of  development  (from 
concept  exploration  to  operational)  and  of  every  type  (mission-critical,  cross-functional, 
infrastructure,  and  administrative).  Investments  which  are  purely  research  and  development 
should  be  considered  part  of  the  organization’s  R&D  portfolio,  rather  than  its  IT  investment 
portfolio. 

For  each  phase  and  type  of  investment,  there  must  be  developed  appropriate  review  processes, 
documentation  requirements,  and  selection  criteria. 

Dollar  thresholds  must  be  established  to  assign  investment  decisions  to  the  appropriate  level 
authority,  but  consistent  decision-making  processes  should  be  used  throughout  the  organization. 
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There  must  be  supplemental  criteria  to  identify  mission-critical  projects  that  fall  below  the  dollar 
threshold  but  still  require  higher  management  review. 

2.3  Selection  Phase: 

The  selection  phase  creates  a  portfolio  of  IT  investments  designed  to  improve  overall 
organizational  performance.  It  requires  a  standard  set  of  criteria  to  judge  which  proposed 
investments  represent  the  best  balance  of  costs,  benefits,  and  risks.  A  consistent,  objective, 
methodology  for  determining  these  performance  measures,  ensuring  they  reflect  organizational 
goals  and  objectives,  and  tracking  them  throughout  the  investment  process  is  the  only  means  by 
which  sound,  rational  investment  decisions  can  be  made. 

Key  Enterprise-level  management  tools  and  techniques  applicable  to  the  selection  phase  include: 

1.  An  executive  management  team  that  makes  funding  decisions  based  on  objective 
comparisons  and  trade-off  between  competing  projects. 

2.  Documented  decision  criteria  that  examine  ROI,  technical  risks,  program  effectiveness, 
customer/user  impact,  project  size  and  scope. 

3.  Pre-defined  thresholds  and  authority  levels  that  put  investment  decisions  in  the  right 
hands. 

4.  Minimal  acceptable  ROI  values  to  minimize  risk  and  increase  returns. 

5.  Risk  assessments  to  expose  potential  technical  and  managerial  weaknesses. 

The  selection  process  has  four  steps: 

2.3.1  Screen  IT  project  proposals 

A  mature  investment  screening  process  will  prescribe  the  amount  and  rigor  of  supporting 
documentation  for  IT  project  proposals  based  on  their  type  and  phase  of  implementation. 
Mission-critical  projects  will  receive  more  detailed  scrutiny  than  less  strategically  important 
ones.  Key  questions  to  be  answered  in  the  screening  process  include: 

1.  Is  the  project  clearly  relevant  to  mission  priorities  outlined  in  the  organization’s  strategic 
plan? 

2.  Is  the  project  feasible  to  design  and  execute,  given  the  agency’s  proven  capabilities? 

3.  Are  commercial  off-the-shelf  systems  available  to  meet  all/most  of  the  project’s  goals? 

4.  Have  other  agencies  done  this  type  of  project?  Have  their  lessons  learned  been 
incorporated  into  project  planning?  Have  re-use  of  their  product(s)  been  considered? 

5.  Does  the  project  conform  to  the  organization’s  technology  and  information  architecture? 
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6.  Will  the  project  be  executed  in  well-defined  stages,  with  clear  decision  points  for 
continuing,  modifying,  or  canceling  the  project? 

2.3.2  Analyze  risks,  benefits,  and  costs 

A  detailed  evaluation  of  each  proposal  is  conducted  and  summarized  to  support  senior 
management’s  decision-making  process.  A  technical  review  team  should  evaluate  the  project’s 
benefit-cost  and  risk  analyses,  and  particularly  the  projected  benefits  to  mission  accomplishment 
and  the  proposed  performance  measures  for  comparing  expected  versus  actual  results.  Key 
questions  in  this  analysis  include: 

1.  Has  the  relevant  office  successfully  managed  other  projects  of  similar  risk  and 
complexity? 

2.  Have  project  risks  been  assessed  using  a  well-defined,  documented  process?  Has  a 
sensitivity  analysis  been  done  for  critical  variables? 

3.  Is  there  a  specific  plan  for  monitoring,  managing,  and  mitigating  project  risks? 

4.  What  are  the  operational  risks  to  users/customers  if  the  project  does  not  proceed? 

5.  Have  users  and  customers  validated  the  proposed  mission  benefits  of  the  project? 

6.  Has  a  systematic,  performance-oriented,  detailed  cost-benefit  analysis  been  prepared? 

7.  What  are  the  constraints  and  assumptions  that  may  affect  the  benefits  of  alternative 
solutions? 

8.  Does  the  justification  for  the  project  depend  on  projected  long-term  (>5  years)  benefits? 

If  so,  what  is  the  level  of  confidence  in  those  projections? 

9.  Do  the  assumptions  supporting  the  analysis  accurately  reflect  market  trends  in  hardware 
and  software?  Do  projected  costs  reflect  today’s  prices  or  those  expected  in  the  execution 
years? 

10.  Are  benefits  clearly  expressed  in  terms  of  improved  mission  performance? 

11.  Can  project  cost  be  shared  with  other  agencies  with  similar  needs? 

2.3.3  Prioritize  projects  based  on  risk  and  return 

After  analyzing  all  proposed  projects,  the  organization  should  use  expected  risks  and  benefits  to 
identify  candidate  investments  with  the  greatest  chances  of  effectively  and  efficiently  supporting 
key  mission  objectives  within  budget  constraints.  It  is  essential  in  making  this  prioritization  that 
all  projects  be  measured  against  a  consistent  set  of  objective,  results-oriented  performance 
measures.  Typical  risk  and  return  criteria  used  in  this  step  include: 
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1.  Investment  Risk.  How  large  is  the  proposed  IT  investment  cost,  particularly  in  comparison  to 
the  overall  IT  budget? 

2.  Project  Longevity  and  Scope.  Is  the  project  using  a  modular  approach?  Is  it  as  narrow  in 
scope  and  duration  as  possible? 

3.  Technical  Risk.  How  will  the  proposed  technology  integrate  with  existing  systems?  Does  the 
project  take  advantage  of  COTS  products?  How  complex  is  the  system  architecture  and 
software  design? 

4.  Mission  Impact.  How  will  the  IT  investment  support  improved  performance  in  specific 
outcome-oriented  terms? 

5.  User/Customer  Needs.  How  well  does  the  investment  address  identified  needs  of  the  IT  user 
or  customer  communities? 

6.  Return  on  Investment  (ROI).  Is  the  calculated  ROI  adjusted  for  risk  and  analytically  sound? 

7.  Organizational  Impact.  What  will  be  the  structural  and  procedural  impacts  of  the 
investment  on  the  organization? 

8.  Expected  Improvement.  Does  the  investment  represent  a  new  capability  or  the  enhancement 
of  existing  ones?  Is  it  mandated  by  law  or  executive  directive?  Is  it  required  to  maintain 
mission-critical  functions?  What  is  the  expected  magnitude  of  the  improvement  in 
performance? 

The  outcome  of  this  step  should  be  a  prioritized  list  of  IT  investments  with  supporting 
documentation  and  analysis.  Typically  the  list  would  sort  out  into  three  groups: 

1.  Likely  winners ,  with  high  returns  and  low  risk. 

2.  Likely  dropouts ,  with  higher  risk  and  low  return. 

3.  Projects  that  warrant  further  study ,  where  risks  and  return  are  more  evenly  balanced. 

The  analytical  and  management  focus  should  be  on  this  group. 

2.3.4  Determine  the  right  mix  of  projects 

Once  the  proposed  investments  are  prioritized,  senior  management  must  make  the  final  selection 
based  on  technical  soundness  of  projects,  their  contribution  to  mission  needs,  performance 
improvement  priorities,  and  IT  funding  levels.  Consideration  must  be  given  to  the  factors: 

1.  The  need  for  strategic  improvements  vs.  keeping  current  systems  operational.  Managers 
must  strike  a  balance  between  continuing  to  invest  in  older  systems  vs.  replacing  them. 
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2.  New  projects  vs.  ongoing  projects.  Projects  approved  for  funding  must  be  periodically 
reviewed  to  ensure  they  should  still  be  supported.  Problems  in  project  execution,  or 
changes  in  mission  or  environment  may  make  new  investments  more  consistent  with 
organizational  objectives. 

3.  High  vs.  low  risk.  Senior  management  must  carefully  balance  the  amount  of  risk  in  the  IT 
portfolio  against  the  organization’s  capabilities  and  ability  to  manage  risk. 

4.  The  impact  of  one  project  on  others.  The  integration  of  systems  means  that  most  new 
initiatives  will  affect  or  be  affected  by  other  projects  or  existing  systems.  Managers  must 
recognize  these  dependencies  and  the  risks  they  generate  and  make  decisions  accordingly. 

5.  The  opportunity  costs  of  funding  or  not  funding  current  proposals.  Too  large  a 
commitment  to  current  investments  may  leave  the  organization  unable  to  take  advantage 
of  future  opportunities.  Conversely,  failure  to  acquire  needed  IT  infrastructure  can 
severely  limit  future  projects.  Managers  must  carefully  judge  trends  in  technology  and 
funding  to  make  the  best  choices. 

6.  External  control  of  funding.  Where  funding  for  all  or  part  of  a  project  is  coming  from 
outside  the  organization,  the  risks  of  this  lack  of  control  must  be  weighed  in  making 
investment  choices. 

7.  Budget  constraints.  DoD  funding  is  highly  dependent  on  economic  and  political  trends 
outside  the  Department’s  control.  Careful  analysis  of  likely  funding  levels  and  the 
possibility  of  external  funding  must  enter  into  the  investment  selection  calculation. 

The  selection  of  the  right  mix  of  IT  investment  projects  with  their  performance  measurement  and 
review  plans,  risk  mitigation  strategies,  and  cost-benefit  analyses  leads  directly  into  the  control 
phase  of  the  IT  investment  management  process. 

2.4  Control  Phase: 

The  control  phase  represents  the  classical  use  of  performance  measures  to  track  cost,  schedule, 
and  performance  against  a  contractual  requirement.  The  critical  flaw  in  many  past  IT  projects 
has  been  the  failure  to  effectively  ground  the  performance  goals  and  objectives  of  a  given  project 
in  any  organizational  reality.  Managers  at  every  level  must  ensure  that  the  focus  of  every  IT 
project  is  to  deliver  the  capabilities  that  the  end  user  requires.  Performance  measures  for  each 
required  capability  must  be  clearly  established  up  front  in  the  requirements  definition  process, 
and  user  involvement  must  be  maintained  all  the  way  to  completion  to  ensure  that  the  mission 
focus  is  not  diluted  as  the  project  progresses. 

Key  Program/Project  Level  management  tools  and  techniques  applicable  to  the  control  phase 
include: 

1.  Processes  that  involve  senior  management  in  ongoing  project  reviews  and  force  decisive 
steps  to  resolve  problems  early  in  the  project. 
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2.  Explicit  measures  and  data  to  monitor  expected  vs.  actual  cost,  schedule  and  performance 
outcomes.  These  must  be  consistently  maintained  throughout  the  organization  and 
readily  available  to  decision-makers  through  automated  management  information 
systems. 

3.  Positive  incentives  for  identifying  real  and  potential  problems  for  management  attention 
and  action. 

The  control  phase  has  two  steps: 

2.4.1  Monitor  actual  vs.  expected  performance 

Managers  at  all  levels  must  monitor  the  progress  of  IT  investment  projects  toward  their  projected 
mission  benefits.  Project  managers  will  monitor  their  projects  continuously,;  senior  managers 
will  require  periodic  reporting  and  project  reviews.  The  key  at  all  levels  is  to  use  a  consistent  set 
of  objective,  outcome-oriented  performance  measures  to  ensure  that  the  right  things  are  being 
measured  and  that  problems  are  identified  as  early  in  the  process  as  possible.  Key  questions  that 
this  step  should  answer  are: 

1.  How  do  current  cost,  schedule,  and  performance  values  compare  with  those 
budgeted/scheduled?  What  are  the  causes  of  any  variances? 

2.  Would  we  fund  this  project  as  a  new  start  today? 

3.  Have  new  requirements  “crept”  into  the  project?  Have  operational  needs  changed  since 
the  project  started? 

4.  Is  the  project  still  technically  feasible?  Is  it  consistent  with  our  standards  and  IT 
architecture? 

5.  How  does  this  project  interact  with  other  projects? 

2.4.2  Taking  action  to  correct  deficiencies 

The  decision  to  continue,  modify,  or  cancel  an  IT  investment  project  should  be  a  deliberate 
management  decision,  documented  and  justified  by  the  review  and  analysis  in  the  previous  step. 
IT  managers  must  ensure  that  operational  users  are  fully  integrated  into  this  decision  process. 

2.5  Evaluation  Phase: 

The  evaluation  phase  takes  place  after  the  IT  project  is  delivered  to  the  user  and  is  operational.  It 
provides  the  final  test  of  whether  the  investment  provides  the  promised  return  in  value  to  the  user 
and  the  organization.  Performance  measurement  provides  the  criteria  by  which  success  is 
measured  and  the  process  by  which  these  criteria  are  evaluated.  Failure  of  an  investment  to 
achieve  the  projected  benefits  may  require  modification  or  termination  of  the  project,  an  overhaul 
of  the  performance  measurement  procedures  of  the  organization,  or  both. 
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Key  Functional  and  Program/Project  Level  management  tools  and  techniques  applicable  to  the 
evaluation  phase  include: 

1.  Post  implementation  reviews  to  determine  actual  project  cost,  benefits,  risk,  and  returns. 

2.  Maintaining  accountability  for  project  performance  and  success  based  on  quantifiable 
measures  and  positive  management  incentives. 

3.  Modification  of  selection  and  control  processes  to  reflect  lessons  learned  and  ensure 
continuous  improvement. 

The  evaluation  phase  has  three  steps: 

2.5.1  Conduct  post  implementation  reviews 

Post  implementation  reviews  should  determine  the  actual  vs.  anticipated  results  of  an  IT 
investment  by  answering  these  questions: 

1.  How  effective  was  the  project  in  meeting  the  original  cost,  schedule  and  performance 
objectives? 

2.  What  operational  benefits  did  the  project  deliver?  Did  they  match  the  projected  ones? 
Why  not? 

3.  Were  the  operational  requirements  and  assumptions  that  justified  the  system  valid? 

4.  What  lessons  can  be  learned  from  this  project? 

2.5.2  Decide  on  adjustments 

Based  on  the  answers  to  the  review,  management  must  decide  whether  to  continue,  modify,  or 
replace  the  operational  system.  The  review  should  contain  analysis  of  the  possible  alternatives 
with  recommendations  on  which  is  the  best  course  to  provide  the  required  operational  capability 
to  the  users. 

2.5.3  Identify  and  implement  lessons  learned 

Using  the  collected  results  of  post  implementation  reviews  across  many  systems,  managers  can 
identify  and  correct  systemic  weaknesses  in  their  organization’s  procedures,  processes,  and 
structure.  Additional  training  of  IM/IT  personnel,  improved  management  control  systems,  and 
more  rigorous  requirements  definition  procedures  are  examples  of  corrective  actions  that  may  be 
needed. 

2.6  Performance  measurement  is  critical 

As  can  be  seen  above,  each  of  these  three  phases  of  the  IT  investment  management  process 
depends  on  the  definition,  collection,  and  evaluation  of  effective  performance  measures. 
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The  remainder  of  the  guide  will  provide  tools,  techniques,  and  guidance  on  how  to  accomplish 
these  critical  tasks. 
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3.  DoD  IT  PERFORMANCE  MEASUREMENT  OVERVIEW 


This  section  presents  the  basics  of  DoD  IT  performance  measurement  to  ensure  a  common 
understanding  throughout  the  Department  including  a  common  definition,  a  common 
understanding  of  how  to  measure  performance,  and  an  explanation  of  the  various  IT 
performance  measurement  levels. 

3.1  Definition 

DoD  defines  IT  performance  measurement  as: 

the  assessment  of  effectiveness  and  efficiency  of  IT  in  support  of  the  achievement 
of  an  organization ’s  missions,  goals,  and  quantitative  objectives  through  the 
application  of  outcome-based,  measurable,  and  quantifiable  criteria,  compared 
against  an  established  baseline,  to  activities,  operations,  and  processes. 

This  definition  has  several  important  components.  IT  investment  performance  measures  must  be 
linked  to  an  organization’s  missions,  goals,  and  quantitative  objectives.  More  precisely  DoD  IT 
investments  must  be  linked  to  the  organization’s  overall  mission  through  the  use  of  performance 
measures.  Other  noteworthy  components  of  the  definition  are  that  the  performance  measures 
must  be  quantifiable,  measurable,  and  compared  against  an  established  baseline. 

3.1.1  Parameters  of  Performance  Measurement 

Performance  measurement  is  the  process  whereby  an  organization  establishes  the  parameters  of 
performance  within  which  programs,  projects,  and  acquisitions  are  obtaining  the  desired  results 
in  support  of  mission  goals.  These  parameters  include: 

•  the  “As-Is”  or  baseline  condition,  which  is  the  level  of  performance  before  the  current 
program,  project,  or  acquisition; 

•  the  current  level  of  performance  achieved  by  the  project  effort; 

•  a  benchmark,  which  is  the  level  of  performance  observed  from  studies  of  best  practices; 

•  the  target  (goal)  for  the  desired  level  of  performance,  frequently  based  on  benchmarks; 
and 

•  the  threshold,  which  is  the  level  of  performance  below  which  the  program,  project,  or 
acquisition  is  no  longer  achieving  acceptable  results. 

3.1.2  Effectiveness  and  Efficiency 

We  evaluate  performance  by  two  criteria:  effectiveness  and  efficiency.  Effectiveness 
demonstrates  that  an  organization  is  doing  the  right  things;  efficiency  demonstrates  that  an 
organization  is  doing  things  optimally. 

3.1.2.1  Effectiveness 

Effectiveness  is  doing  the  RIGHT  things. 

•  Achievement  of  missions  and  goals 
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•  Customer  satisfaction 

•  Quality  of  work 

Important  effectiveness  questions  are: 

•  Has  the  organization  achieved  its  missions  and  goals? 

•  Are  end  users  of  its  products  and  services  satisfied  customers? 

•  Was  the  work  of  high  quality? 

3.1.2.2  Efficiency 

Efficiency  is  doing  things  by  employing  the  BEST  use  of  available  resources. 

•  Quantity  of  work 

•  Cost  of  work 

•  Timeliness  of  delivery  (schedule) 

Typical  efficiency  measures  relate  to  inputs,  outputs,  and  processes,  and  might  include  the 
following  questions: 

•  Do  obligation  rates  match  the  annual  budget? 

•  Was  the  IM/IT  effort  completed  on  time  and  on  budget? 

•  How  much  of  the  product  and  service  was  produced? 

•  How  many  employees  or  full-time  equivalents  (FTEs)  were  required? 

Evaluation  of  a  program's  effectiveness  and  efficiency  begins  with  the  establishment  of  a 
performance  measurement  baseline.  Performance  measures  are  developed  based  on  expected 
outcomes,  assessed  against  the  baseline,  and  continually  monitored  to  determine  whether  they  are 
being  achieved.  Individual  measures  are  defined  and  then  quantified  with  targets  and  thresholds 
to  form  the  performance  measurement  baseline. 

3.1.3  Steps  for  Measuring  DoD  IT  Performance 

This  section  briefly  highlights  the  generic  steps  for  measuring  the  IM/IT  performance  of  an 
organization,  program,  or  project. 

3.1.3.1  Step  1  Define  mission,  key  result  areas,  and  business  functions 

•  Why  does  the  organization  exist?  (MISSION) 

•  What  major  programs  are  performed  by  the  organization? 

•  What  work  effort(s)  support  major  programs? 

•  What  are  specific  RESULTS  produced/delivered  by  each  work  effort? 

•  Who  are  its  customers? 

•  What  are  customer  and  provider  expectations? 

•  What  are  core  competencies? 

3.1.3.2  Step  2  Develop  mission  related  goals 

•  Are  there  standards/goals  associated  with  the  mission  ? 

•  Are  historic  data  available  upon  which  to  base  goals? 

•  Are  data  accurate  and  reliable? 
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•  Are  performance  goals  realistic? 

•  Do  performance  goals  represent  increased  efficiency  and  effectiveness? 

•  Will  performance  goals  yield  improvement  in  one  or  more  Key  Result  Areas? 

•  How  can  we  identify  and  adapt  the  best  practices  to  improve  organizational 
performance  (i.e.,  benchmarking)? 

•  How  does  the  approach  compare  to  best  practices  in  the  industry? 

3.1.3.3  Step  3  Generate  performance  measures 

•  What  is  our  product/service? 

•  Is  it  measurable? 

•  What  unit/scale  of  measure  is  appropriate? 

•  Which  measurable  criteria  have  meaning  to  whom? 

•  What  is  the  performance  measurement  (units  &  equations)? 

•  What  Key  Result  Area  does  the  performance  measure  characterize? 

3.1.3.4  Step  4  Validate  and  verify  performance  measures 

•  Does  the  measure  provide  useful  and  important  information  on  the  program  that 
justifies  the  difficulties  in  collecting,  analyzing  or  presenting  the  data? 

•  Does  the  measure  address  the  aspect  of  concern?  Can  changes  in  the  value  of  the 
measure  be  clearly  interpreted  as  desirable  or  undesirable?  Is  there  a  sound,  logical 
basis  for  believing  that  the  program  can  have  an  impact  on  the  measure? 

•  Does  the  information  provided  by  the  measure  duplicate  or  overlap  with  information 
provided  by  another  measure? 

•  Are  likely  data  sources  sufficiently  reliable  or  are  there  biases,  exaggerations, 
omissions,  or  errors  that  are  likely  to  make  the  measure  inaccurate  or  misleading? 

•  Can  data  be  collected  and  analyzed  in  time  for  the  decision? 

•  Are  there  concerns  for  privacy  or  confidentiality  that  would  prevent  analysts  from 
obtaining  the  required  information? 

•  Can  the  resource  or  cost  requirements  for  data  collection  be  met? 

•  Does  the  final  set  of  measures  cover  the  major  concerns? 

•  Are  we  measuring  the  right  things? 

3.1.3.5  Step  5  Implement  the  performance  measures  and  collect  data 

•  Is  the  data  accessible  across  tiers? 

•  Are  we  prepared  to  manage  cultural  change  within  the  organization? 

3.1.3.6  Step  6  Monitor  and  assess  the  results  and  repeat  the  process  as  needed 

•  Can  we  measure  better  because  of  our  analyzed  results? 

•  How  can  we  improve  our  business  processes? 

•  How  should  goals  be  used  to  improve  resource  efficiency  and  customer  deliverables? 

•  Are  current  Key  Results  Areas  adequate? 

•  What  recommendations  should  be  forwarded  to  or  acted  upon  by  appropriate  tiers. 
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3.2  Users  Of  Performance  Measurement  Information 

There  is  a  recognition  that  different  management  tiers  need  different  kinds  of  information  to 
make  business  and  investment  decisions.  Many  different  audiences  are  interested  in  the 
performance  of  your  program,  project,  or  acquisition.  Your  set  of  performance  measures  must 
meet  the  requirements  of  all  of  these  audiences. 

At  all  levels  of  review,  managers  should  evaluate  what  they  are  doing  and  how  productively  they 
are  doing  it.  Program  problems  should  not  be  surprises.  Managers  want  to  know  whether  a 
program  is  on  track  or  at  risk.  If  a  program  is  not  on  track,  managers  want  to  know  that  problems 
have  been  resolved  and  that  the  program  will  meet  desired  objectives. 

There  are  three  tiers  or  levels  of  performance  measurement  users  within  DoD,  as  shown  in 
exhibit  3-1:  As  you  move  down,  the  level  of  performance  detail  increases.  Measures  always 
align  upward,  not  top  down. 


Measure 

A  \  Relationships 

ENTERPRISE 

J  /  Executive^  V 

M  /x  e  ,.  \  \  Policy  and  mission 

r  /  InlormationX  \ 

/  /,  ,•  •  „  ,A  \  decisions  and  strategies 

/  /  Mission  Results\  \  .  .... 

/  /  \  \  Accountability 

Timing 

Cyclical 

J?  /  M  \  \ 

&  /  /  \  \  A,  Management  and 

£  /  /  FUNCTIONAL  \\  %  improvement  of 

/  /  Management  \  \  ^  operations 

~  /  /  Information  \\  o  Integration  & 

/  /  Unit  Results  \  \  planning 

Periodic 

/  /  PROGRAM/PROJECT  \\  Tactical  and 

/  /  Activity/Task  \  \  _  execution 

f  /  Information  \  >  management 

/  Workplace  Results  \  Resource 

/  \  allocation 

Immediate 

Exhibit  3-1:  Levels  of  Performance  Measurement 


3.2.1  Enterprise  Level. 

At  the  enterprise  level,  the  focus  is  on  mission  results,  and  information  is  needed  to  choose 
policy  directions  and  make  mission  decisions.  Managers  make  the  connection  between  the  IT 
performance  measurement  requirements  of  TTMRA  and  the  overall  organizational  performance 
measurement  required  by  GPRA.  IT  investment  decisions  are  greatly  influenced  by  this  level 
during  the  Selection  phase  of  the  investment  process.  The  timing  for  information  is  cyclical. 

Relationship:  Policy,  mission  decision  and  strategies 

Role:  Accountability 


3-4 


Timing:  Cyclical 

These  managers  are  responsible  for  reporting  and  justifying  the  use  of  IM/IT  expenditures  to 
Congress,  OMB,  the  General  Accounting  Office  (GAO),  and  other  external  entities.  Examples  of 
these  managers  are  senior  DoD  officials,  such  as  the  Secretary  of  Defense,  the  Deputy  Secretary 
of  Defense,  Chairman  of  the  Joint  Chiefs  of  Staff,  Undersecretaries  of  Defense,  Assistant 
Secretaries  of  Defense,  the  heads  of  the  Military  Departments,  CIO  of  the  DoD,  Principal  Staff 
Assistants  (PSAs),  and  the  Combatant  Commanders  in  Chief  (CINCs)..  At  the  enterprise  level, 
leaders  consider  major  policy  questions: 

•  Are  investments  in  IT  yielding  acceptable  return  on  investment  (ROI),  including 
quantifiable  improvements  in  mission  effectiveness? 

•  Are  dollars  invested  in  IT  yielding  the  expected  results? 

•  Are  investment  priorities  synchronized  with  overall  DoD  mission  priorities? 

•  Are  approved  IT  architectures  being  implemented  in  a  timely  and  cost-effective  manner? 

•  Is  there  a  proactive  oversight  system  to  ensure  benefit,  cost,  and  schedule  goals  are  met? 

•  Is  the  IT  strategic  plan  explicitly  linked  to  the  functional,  component,  and  departmental 
strategic  plans? 

3.2.2  Functional  Level 

At  the  Functional  Level,  the  focus  is  on  unit  results  where  information  is  needed  to  manage  and 
improve  operations.  These  managers  are  responsible  for  reporting  the  performance  of  major 
DoD  functions  across  multiple  projects,  programs,  or  acquisitions.  They  combine,  synthesize 
and  report  program/project-level  results  for  use  by  enterprise-level  managers.  This  level  is 
heavily  involved  in  the  Selection  and  Evaluation  phases  of  the  investment  process.  The 
program/project  level  serves  as  the  Functional  Level  representative  during  the  Control  phase  of 
the  investment  process.  The  timing  for  information  is  periodic. 

Relationship:  Management  and  improvement  of  operations 
Role:  Integration  &  Planning 

Timing:  Periodic 

Functional-level  managers  include  those  people  who  report  directly  to  the  PSAs,  the  CIOs  of  the 
Military  Departments,  and  the  Service  Acquisition  Executives.  These  managers  consider  such 
questions  as: 

•  Do  efforts  under  my  oversight  help  achieve  DoD  strategic  objectives? 

•  Are  funded  efforts  synchronized  over  time?  For  example,  if  Project  A  feeds  into  Project 
B,  are  they  properly  resourced  and  on  schedule  to  achieve  proper  integration? 

•  Are  best  practices  and  lessons  learned  being  applied  across  all  programs? 

•  Are  related,  functional  efforts,  such  as  functional  process  improvements  or  infrastructure 
investments,  resourced  properly  and  linked  to  outputs  of  my  IT  project? 

•  Are  performance  measures  being  tracked  and  put  into  reports  that  reflect  DoD 
performance  required  by  Congress? 
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•  Do  mission  support  processes,  such  as  business  process  reengineering  (BPR),  migration 
systems,  and  data  standardization,  support  effective  accomplishment  of  the  DoD  mission? 

The  functional  level  is  also  where  the  interests  of  the  IT  user  community  are  directly  represented. 
It  is  at  the  functional  level  that  requirements  of  the  users  of  IT  systems  are  approved, 
documented,  funded,  and  provided  to  the  program/project  level  for  execution.  The  functional 
level,  in  concert  with  the  users,  is  also  responsible  for  conducting  post-implementation 
assessments  of  IT  investments’  operational  capabilities,  to  ensure  that  the  expected  functional 
benefits  of  an  investment  are  actually  realized.  This  post-deployment  assessment  is  crucial  to 
ensuring  that  the  linkage  between  IT  investments  and  organizational  performance  is  maintained. 

3.2.3  Program/project  level 

At  the  program/project  level,  activity  and  task  information  is  critical  to  make  tactical  decisions 
and  execute  management  decisions.  Managers  are  leading  programs,  projects,  or  acquisitions 
that  are  sponsored  by  functional-level  managers.  These  managers  are  involved  in  the 
accomplishment  of  actual  IM/IT  efforts.  The  program/project  level  addresses  the  expected 
outcomes  and  results  of  IT  investments.  This  level  involves  the  collection  of  information 
concerning  the  outcome/result  of  the  IT  investment’ s  performance  and  the  comparison  of  this 
performance  against  the  established  baseline  for  that  investment.  Program/project  level  measures 
are  combined,  synthesized  and  reported  to  the  functional  level.  This  level  serves  as  the 
representative  of  the  Functional/user  level  during  the  Control  and  Evaluation  phases  of  the 
investment  process.  The  timing  for  information  is  immediate. 

Relationship:  Tactical  and  execution  management 
Role:  Resource  allocation 

Timing:  Immediate 

These  managers  consider  such  questions  as: 

•  How  does  my  effort  contribute  to  my  sponsor’s  strategic  and  tactical  objectives? 

•  Is  my  effort  within  budget? 

•  Is  my  effort  on  schedule? 

•  Is  my  effort  meeting  the  specified  functional  and  performance  requirements? 

It  is  possible  for  an  individual  to  act  in  the  capacity  of  multiple  levels.  For  example,  an 
individual  whose  primary  responsibility  is  program  management  of  a  functional  area  may,  at 
various  times,  actually  lead  a  project.  In  managing  a  particular  IM/IT  program,  project,  or 
acquisition,  the  individual  is  working  as  a  project  manager. 

Exhibit  3-2  illustrates  the  reporting  relationships  among  the  external  oversight  agencies  and  the 
enterprise,  program,  and  program/project  levels.  Guidance  related  to  expected  results  flows 
down  the  chain  of  command  to  the  project  managers  who  are  responsible  for  execution  of 
individual  IT  initiatives. 
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Exhibit  3-2:  Reporting  Requirements 

Most  performance  data  are  collected  at  the  program/project  level.  Program/project-level 
managers  use  performance  measures  to  make  decisions  for  their  program,  project,  or  acquisition. 
The  program/project-level  managers  then  provide  data  on  their  programs  to  managers  at  the 
program  level.  Functional-level  managers  collect  and,  in  some  cases,  summarize  the 
program/project-level  results  to  make  managerial  decisions  across  programs,  projects,  or 
acquisitions  within  a  functional  area.  Functional-level  managers  provide  reports  and 
recommendations  to  the  enterprise  managers,  who  use  the  information  to  make  strategic 
decisions. 

Enterprise  managers  then  report  the  information  to  Congress,  stakeholders,  and  other  external 
oversight  interests,  such  as  OMB  and  GAO,  as  a  basis  to  justify  future  IM/IT  investments. 


3-7 


4.  PERFORMANCE  MEASUREMENT  AT  THE  ENTERPRISE  LEVEL 


This  section  contains  guidance  for  developing  and  using  performance  measures  at  the  enterprise 
level .  The  approach  used  within  DoD  focuses  on  these  five  key  principles: 

Establish  Hierarchies  of  Goals.  GPRA  and  ITMRA  requires  clear  organizational  hierarchies  of 
goals  and  performance  measures  be  established.  To  comply  with  the  spirit  and  intent  of  both 
laws,  the  goals  and  measures  used  at  lower  organizational  levels  should  be  linked  with  DoD’s 
mission/strategic  goals.  Research  has  indicated  that  without  clear  hierarchically  linked  goals  and 
performance  measures,  managers  and  staff  throughout  the  organization  lack  straightforward 
roadmaps  showing  how  their  work  contributes  to  attaining  organizational  strategic  goals. 

If  an  IT  investment  does  not  measurably  improve  agency  mission  performance  (  no  matter  how 
well  the  program  met  its  cost  and  schedule  baselines  or  output  and  performance  indicator 
measures)  that  investment  should  not  be  made. 

Measure  Goal  achievement.  Measures  are  applied  at  many  levels.  A  few  well-chosen,  outcome- 
oriented  measures  are  better  than  multiple,  potentially  conflicting,  suboptimal  measures. 

Empower  the  field.  Empowerment  means  making  the  factors  for  incentives  and  disincentives  of 
actions  and  decisions  visible.  Performance  measures  allow  managers  at  all  levels  to 
measure  performance  and  compare  actual  results  against  stakeholder  expectations  for  local 
use. 

Find  Trend  Indicators.  Indicators  will  be  selected  to  measure  progress  towards  a  particular  goal. 
The  user  should  be  able  to  graph  value  with  respect  to  time  for  quality,  quantity,  etc., 
illustrated  below.  Each  value  is  represented  as  a  "high-low-most  likely"  to  represent  the 
range  of  responses.  Note  that  completion  of  an  action  is  not  a  trend. 

Measure  Outcomes.  Measures  are  typically  categorized  as  input,  output,  and  outcome.  Input 

measures  are  relatively  easy  to  quantify  and  capture,  e.g.  resources,  requests,  students,  etc. 
Output  measures  can  be  quantified  for  organizations  with  formal  product  and  service 
descriptions  but  difficult  for  those  with  more  abstract  mission  statements.  Outcome 
measures  of  the  vision  or  stakeholder  satisfaction  with  products  and  services  are  multi¬ 
dimensional  and  hard  to  identify  and  quantify.  However  complex,  only  outcome  measures 
are  considered  worth  pursuing  at  the  enterprise  and  functional  levels. 

Engage  Stakeholders.  Stakeholders  have  to  be  defined  at  each  organizational  level  and  include 
the  following: 

•  customers  and  suppliers  -  current  and  future 

•  employees  and  support  contractors  -  current  and  future 

•  higher  order  management  -  e.g.  headquarters,  OSD,  OMB,  Congress 

•  subordinate  management  -  e.g.  headquarters-field,  OMB-OSD,  OSD-MILDEPS,  etc. 
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4.1  The  IT  Strategic  Plan 

Performance  measurement  at  this  level  starts  with  the  IM/IT  Strategic  Plan.  This  defines  the 
mission,  goals,  and  objectives  of  the  organization  and  how  IM/IT  programs  and  initiatives 
support  the  accomplishment  of  these.  Depending  on  the  size  and  mission  of  the  organization, 
these  goals  and  objectives  can  be  quite  broad,  and  performance  measures  must  be  developed  and 
used  very  carefully  to  be  truly  objective  and  meaningful. 

4.1.1  Mission  Statement 

The  first  step  in  defining  IM/IT  performance  measures  at  the  enterprise  level  is  to  define  the 
mission  of  the  IM/IT  function  within  the  organization.  This  is  usually  derived  from  these 
sources: 

1.  The  overall  mission  statement  of  the  organization. 

2.  Policy  directives  and  guidance  from  the  organizational  leadership. 

3.  External  sources  (legislation,  OMB  directives,  etc.) 

An  example  is  the  DoD  IM/IT  mission  statement: 

“ Provide  the  right  information,  at  the  right  time,  to  the  right  destination,  in  a  form  that  users  can 
understand  and  reliably  use  to  accomplish  their  objectives.  ” 

This  is  an  exceptionally  broad  mission  statement,  because  of  the  extensive  scope  of  the  DoD 
IM/IT  function  and  the  organizational  structure  of  DoD.  Smaller  organizations  with  a  greater 
focus  on  specific  IM/IT  functions  will  have  more  specific  and  narrow  missions. 

4.1.2  Goals 

From  the  mission  statement  the  strategic  planner  defines  specific  goals  that  must  be  achieved  in 
order  to  accomplish  the  stated  mission.  These  goals  are  at  the  next  lower  level  of  detail  from  the 
mission  statement,  and  may  be  organized  according  to  the  organizational  structure  (each  goal 
representing  a  subordinate  element’s  IM/IT  mission),  by  functional  area  (how  IM/IT  will  support 
specific  aspects  of  the  organization’s  functions),  by  specific  IM/IT  functions  (infrastructure, 
security,  communications,  etc.),  or  by  a  combination  of  these.  The  key  is  to  ensure  that  each  goal 
directly  links  back  to  the  overall  IM/IT  mission  and  to  the  overall  mission  of  the  organization. 

At  this  level,  all  goals  should  be  outcome,  rather  than  output-based.  Only  those  organizations 
with  very  narrowly  defined  missions  as  providers  of  services  and  products  can  effectively  define 
performance  purely  in  output  terms.  For  each  goal,  therefore,  a  measurable  outcome  must  be 
defined. 

Examples  of  goals  and  their  outcomes  at  the  DoD  level  are: 

Goal  1:  Improve  effectiveness  and  reduce  the  cost  of  meeting  military  missions  and  performing 
supporting  business  functions. 

Outcome:  All  DoD  organizations  are  accountable  to  stakeholders  for  cost  and  performance 
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Goal  2:  Establish  and  manage  an  integrated  global,  secure,  affordable,  and  common 
infrastructure. 

Outcome:  Users  and  using  commands  get  information  technology  services  to  satisfy  their 
requirements,  easily  and  in  a  timely  manner,  anywhere  in  the  world.  Infrastructure  planning 
ensures  that  long  lead  time  items,  capacity,  facilities,  and  contracts,  are  available  when  needed. 

Goal  3:  Ensure  the  warfighters  and  those  who  support  them  have  access  to  accurate,  high  quality, 
timely,  and  relevant  information. 

Outcome:  Users  get  information  they  can  understand  and  quick  assistance  to  resolve  problems; 
suppliers  can  easily  make  information  available,. 

Goal  4:  Assure:  (1)  the  information  infrastructures  on  which  DoD  ’s  operations  depend  are 
protected  against  exploitation,  disruption,  and  denied  of  use;  and  (2)  that  best  practices  and 
procedures  are  used  to  protect  and  defend  systems,  to  include  provisions  for  continuity  of 
operations  and  the  restoration  of  critic  cd  serx’ices. 

Outcome:  Users  can  get  authentic,  accurate  information,  and  DoD  information  systems  and 
resources  are  protected  against  information  warfare,  terrorist,  and  criminal  activities.  There  are 
risk  mitigation  provisions  for  restoring  critical  services  and  systems  in  a  priority  manner. 

4.1.3  Objectives 

Objectives  carry  the  process  down  to  the  next  level  of  resolution.  They  are  defined  in  terms  of 
specific,  quantifiable,  and  measurable  outcomes  that  contribute  directly  to  the  achievement  of  the 
organizations’  IM/IT  goals.  Objectives  may  be  tied  to  a  specific  IM/IT  initiative  or  program,  a 
specified  level  of  compliance  with  standards,  or  preferably,  a  measurable  improvement  in 
functional  effectiveness  or  efficiency. 

4.2  Identifying  Outcome  Performance  Measures 

Having  defined  mission,  goals,  and  objectives,  the  next  step  is  to  identify  the  performance 
measures  that  will  define  accomplishment  of  each.  Each  performance  measure  has  to  answer  the 
following  questions: 

•  What  is  the  purpose  of  the  measure? 

•  Who  measures  and  how? 

•  Who  uses  it  and  for  what? 

•  What  is  the  cost  of  measure  vs.  value  to  the  user? 

•  What  tools  and  assistance  are  available  to  collect  and  use  measurement  data? 

•  What  special  provisions  must  be  considered? 

4.2.1  Purpose  of  the  measure. 

The  purpose  of  any  performance  measure  is  to  assess  the  accomplishment  of  the  associated 
objective.  For  example,  an  objective  stated  in  the  DoD  IM/IT  Strategic  Plan  is  to: 
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“ Improve  IT  Acquisition  through  streamlined  contracting,  increased  use  of  COTS  to  shorten 
acquisition  cycle,  and  more  effective  reutilization  of  IT  assets.  ” 

Some  appropriate  performance  measures  associated  with  this  objective  would  be: 

‘  Average  contract  award  time  for  DoD  IT  acquisitions(in  months  after  RFP  release).  ” 

“Percentage  of  DoD  IT  acquisitions  requiring  use  of  COTS.  ” 

At  the  enterprise  level,  it  is  best  to  select  measures  that  can  provide  an  indication  of  long-term  or 
systemic  trends  within  the  organization. 

4.2.2  Who  measures  and  how? 

Which  offices  and  activities  will  collect,  analyze,  verify,  validate  and  track  data  for  performance 
measurement?  What  is  the  measurement  strategy  and  methodology?  Section  8  lists  a  number  of 
performance  measurement  methods  that  are  readily  adapted  at  the  enterprise  level. 

4.2.3  Who  uses  the  data  and  for  what? 

Who  within  the  organization  will  receive  the  performance  measurement  data  and  what  will  they 
do  with  it?  What  internal  reporting  mechanisms  are  in  place  to  handle  the  data?  What  are  the 
external  reporting  requirements? 

4.2.4  What  is  the  cost  of  measure  vs.  value  to  the  user? 

A  cost-benefit  analysis  must  be  applied  against  any  proposed  performance  measurement  system. 
If  the  costs  of  collecting  and  analyzing  the  data  exceeds  the  benefit  to  the  organization,  the 
measure  should  not  be  used,  unless  it  is  mandated  by  a  higher  authority.  Similarly,  the  benefits 
of  the  performance  measurement  effort  must  be  apparent  to  those  performing  the  data  collection 
and  analysis  in  order  to  ensure  accurate  and  complete  data  is  reported. 

4.2.5  What  tools  and  assistance  are  available  to  collect  and  use  measurement  data? 

What  automated  tools  are  available  for  the  performance  measurement  effort?  What,  if  any, 
existing  data  collection  and  reporting  systems  are  already  in  place  that  can  be  use  or  adapted  to 
meet  performance  measurement  needs.  What  specialized  training  is  needed  and  is  it  available 
within  the  organization? 

4.2.6  What  special  provisions  must  be  considered? 

Are  there  any  unusual  constraints  or  requirements  on  the  effort?  For  example,  a  National 
Security  System  (NSS)  IT  project  may  have  security  constraints  that  affect  how  performance 
measurement  data  can  be  collected  and  disseminated. 
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5.  PERFORMANCE  MEASUREMENT  AT  THE  FUNCTIONAL  LEVEL 

5.1  Identify  IT  Effort  and  Its  Mission  and  Objectives 

5.1.1  Analyze  Guidance 

Performance  measurement  at  the  functional  level  begins  with  analysis  of  the  organization’s 
higher  headquarter’s  mission,  functional  strategic  plan(s),  IT  strategic  plan  and  other  related 
guidance.  From  these  sources,  functional  managers  and  commanders  at  the  functional  level 
determine  their  organization’s  mission  and  objectives  and  how  IM/IT  activities  and  initiatives 
support  them. 

5.1.2  Analyze  Functions  and  Processes 

Having  identified  their  functional  goals  and  objectives,  and  the  relationship  to  IM/IT  activities, 
functional-level  managers  must  analyze  the  functions  and  processes  their  organizations  perform 
to  determine  what  adjustments  must  be  made.  An  effective  process  for  accomplishing  this 
analysis  is  through  business  process  reengineering  (BPR). 

Through  BPR,  the  organization  clarifies  its  goals  and  objectives,  analyzes  its  functional 
processes  in  light  of  their  contribution  to  organizational  success,  and  systematically  redesigns 
and  streamlines  those  processes,  and  the  supporting  organizational  structures  and  information 
systems  to  achieve  the  desired  goals. 

ITMRA  specifically  requires  that  BPR  or  a  similar  analysis/redesign  process  be  done  before  any 
new  IT  initiative  is  funded.  There  are  a  number  of  government  and  private  sources  providing 
information  and  assistance  with  BPR;  the  DoD  BPR  World  Wide  Web  (WWW)  Page 
(http://137.246.33.239/links.htm)  provides  access  to  several  of  the  most  useful. 

5.1.3  Identify  Requirements  for  IT  Initiatives 

As  a  result  of  the  functional  process  analysis  and  reengineering  efforts  through  the  organization, 
requirements  to  modify,  replace,  or  add  information  technology  systems  will  be  generated.  These 
requirements  will  flow  up  from  the  functional  user  level,  the  level  which  will  actually  use  the 
required  IT,  to  the  appropriate  approval  authority,  depending  on  the  scope  and  cost  of  the 
requirement. 

5.1.3.1  Mission  Needs 

Broad  operational  needs  are  first  identified  in  a  Mission  Needs  Statement  (MNS)  or  similar 
document  that  describes  the  functional  mission  deficiency,  discusses  the  analysis  performed  to 
support  the  deficiency,  explains  why  process  redesign  or  other  non-materiel  solutions  are 
inadequate  or  not  preferred,  identifies  possible  alternative  solutions,  and  lays  out  key  operational 
constraints  and  conditions.  The  user  must  begin  the  performance  measurement  process  here,  by 
identifying  a  very  high  level,  quantifiable  outcome  measure  for  each  broad  operational 
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requirement  defined  in  the  MNS.  Projected  cost  and  schedule  requirements  are  also  identified  at 
this  stage  as  a  basis  for  further  development. 


5.1.3.2  Operational  Requirements 

As  the  requirements  are  further  refined  and  developed,  the  broad  operational  needs  defined  in  the 
MNS  are  translated  into  specific  performance  requirements  in  the  Operational  Requirements 
Document  (ORD)  or  its  equivalent.  (The  ORD  is  defined  in  Appendix  II,  DoD  Regulation 

5000. 2- R.)  The  ORD  breaks  down  the  broad  requirements  of  the  MNS  into  the  specific 
functional  capabilities  required  to  obtain  the  desired  end  result.  It  also  expands  the  scope  of  the 
requirement  to  include  logistical  support  and  maintenance  requirements,  human  factors 
engineering,  standardization,  interoperability,  and  commonality,  etc.  For  each  requirement 
identified  within  the  ORD,  the  user  must  work  with  the  program/project  manager  to  identify  an 
outcome  measure  and  associated  target  and  threshold  parameters:  the  threshold  being  the 
minimum  acceptable  performance  required  to  meet  the  mission  need;  the  target  being  the  ideal  or 
desired  performance.  Careful  and  accurate  definition  of  these  terms  is  crucial  to  establishing 
effective  measurement  of  performance. 

5.2  Identify  Links  to  Enterprise  Mission  and  Strategic  Goals 

A  key  point  in  ITMRA  is  the  requirement  that  all  IT  investments  must  be  explicitly  linked  to 
supporting  organizational  missions  and  strategic  goals.  At  each  level  of  the  process,  managers 
must  ensure  that  their  goals  and  objectives  directly  support  the  accomplishment  of  these  higher- 
level  priorities.  In  many  cases,  the  linkage  will  be  obvious,  but  as  the  definition  of  requirements 
becomes  more  detailed  the  links  may  become  vague  or  lost  altogether.  A  conscious,  focused 
effort  must  be  made  at  each  stage  of  the  process  to  identify  and  document  the  relationship  of  each 
requirement  to  the  accomplishment  of  a  higher-level  goal  or  objective.  This  is  a  key  step  in 
building  the  business  investment  case  needed  to  obtain  funding. 

The  use  of  the  Work  Breakdown  Structure  (WBS)  provides  a  useful  tool  for  managing  this 
process  and  ensuring  that  these  linkages  are  maintained  and  documented.  DoD  Regulation 

5000.2- R  mandates  the  use  of  the  WBS  as  a  tool  for  managing  performance  measurement  for 
major  acquisition  programs;  it  is  easily  adapted  to  an  overall  organizational  approach  to  defining 
and  linking  mission,  goals,  and  objectives  at  every  level  of  the  organization. 

5.3  Define  IT  Effort  Baseline  and  Develop  Performance  Measurement  Framework 
5.3.1  IT  Effort  Baseline 

At  the  functional  level,  IT  managers  are  concerned  with  measuring  performance  across  multiple 
IT  projects  and  initiatives  and  relating  that  performance  to  their  functional  missions  goals,  and 
objectives  and  to  the  strategic  objectives  of  the  larger  organization.  Development  of  the  baseline 
for  a  functional-level  IT  effort  requires  these  steps: 

1.  Identify  and  document  all  IT  investments  and  their  contribution  to  the  organization’s 
mission  goals  and  objectives 


5-2 


2.  Define  and  document  the  functional  outcome  requirements  for  each  investment. 

3.  Identify/develop  performance  measures  for  each  requirement. 

4.  Determine  performance  measures’  targets  and  thresholds. 


5.3.1. 1  Identify  and  Document  IT  Investments 

IT  investments  are  proposed  or  existing  systems,  functions,  or  processes  that  use  information 
technology  to  support  or  accomplish  an  organization’s  mission.  Examples  of  IT  are  the  data 
warehousing  systems  that  support  a  personnel  center,  the  Combat  Information  Center  on  a 
warship  -  this  would  be  a  National  Security  System  (NSS)  IT  investment  ,  the  IT  that  supports 
an  air  traffic  control  system,  or  the  office  automation  suite  that  support  a  staff  office  in  the 
Pentagon.  A  less  obvious  examples  is  an  initiative  that  does  not  involve  buying  new  technology 
per  se,  such  as  a  systems  migration  program. 

Functional-level  managers  must  take  stock  of  their  current  and  proposed  IT  investments  and 
carefully  review  and  document  how  each  links  to  mission  accomplishment  or  compliance  with  a 
higher-directed  goal  or  objective.  Where  the  link  is  absent  or  is  vague,  the  investment  must  be 
considered  for  termination  or  redesign. 

5.3.1.2  Define  and  Document  the  Functional  Outcome  Requirements 

The  functional  requirements  for  IT  systems  should  be  completely  reflected  in  the  their  respective 
mission  needs  statements  and  operational  requirements  documents.  For  fielded  systems,  a 
review  and  comparison  of  the  original  requirements  must  be  made  in  light  of  changed  missions 
and  differences  in  operating  environments.  Requirements  that  were  critical  five  years  ago  may 
now  be  irrelevant  to  the  organization’s  current  needs.  Where  requirements  have  changed, 
managers  must  assess  the  need  for  modifying  or  replacing  the  affected  system,  or  simply  revising 
the  measurement  criteria. 

For  non-material  processes  and  functions,  requirements  are  usually  more  loosely  documented  and 
less  clearly  defined.  Managers  should  apply  the  same  scrutiny  to  existing  investments  that  they 
would  for  new  ones.  The  questions  that  must  be  answered  are: 

•  How  does  (or  would)  this  investment  support  my  mission? 

•  What  are  the  desired  outcomes  of  this  investment? 

Functional  area  objectives  may  often  be  found  in  the  functional  economic  analysis  (FEA)  or  the 
results  of  a  BPR  project.  The  FEA  may  also  contain  find  benchmark,  target,  and  current 
estimates  for  the  objectives. 

For  example,  DoD  would  like  to  decrease  cost  and  cycle  time  for  providing  goods  and  services  to 
the  warfighters.  The  functional  user  should  know  the  current  or  baseline  cycle  time  for  filling  a 
part  requisition.  They  could  conduct  a  benchmarking  study  of  the  best  practices  of  the  private 
sector  or  other  government  agencies  to  determine  the  achievable  cycle  time.  Benchmarks  give  a 
perspective  on  the  functional  area’ s  alignment  with  comparable  measures  in  comparable 
organizations. 
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Benchmarking  is  a  method  of  obtaining  values  that  are  based  upon  the  best  practices  of  industry 
or  other  government  agencies.  Benchmarking  is  defined  as  searching  for  the  best  practices 
leading  to  superior  performance.  In  The  Benchmarking  Workbook:  Adopting  Best  Practices  for 
Performance  Improvement  (1991),  author  Gregory  H.  Watson  describes  performance 
benchmarking  as  “the  analysis  of  relative  business  performance  among  direct  or  indirect 
competitors.”  The  Benchmarking  Workbook  recommends  a  process  that  involves  planning, 
searching,  observing,  analyzing,  and  adapting  to  gain  improvement  in  business  practices. 

The  functional  area  ROI  is  associated  with  the  benchmark  and  target  values  for  the  functional 
objectives.  ROI  can  be  computed  to  answer  the  question  of  what  did  I  or  what  will  I  get  for  the 
investment.  ROI  is  the  ratio  of  the  present  value  of  the  benefits  achieved  over  the  life  cycle  of  an 
IT  investment  to  the  present  value  of  the  investment  cost  over  the  life  cycle  of  an  IT  investment. 
Consult  the  OSD  (PA&E)  Automated  Information  System  (AIS)  Economic  Analysis  (EA)  Guide, 
PA&E,  1  April  1995  for  more  information  concerning  ROI. 

Many  projects  are  initially  approved  using  an  estimation  of  ROI.  Maintaining  this  ROI  depends 
on  program  efficiency  and  effectiveness.  If  either  part  of  the  equation  changes,  the  ROI  should 
be  recalculated  to  ensure  that  it  is  still  above  the  minimum  target  for  justifying  the  investment. 

The  initial  ROI  value  is  normally  calculated  as  part  of  a  FEA  that  reflects  the  functional  area’s 
strategic  planning  and  BPR  activities.  After  a  baseline  ROI  is  established,  current  ROI  estimates 
are  calculated  during  program  execution  as  costs  and  benefits  become  more  certain.  ROI  should 
be  tracked  to  ensure  that  the  continuation  of  the  effort  still  makes  sense. 

5.3.1.3  Identify/Develop  Performance  Measures  for  Each  Requirement 

The  accomplishment  of  functional  missions  and  goals  (mission  benefit) ,  not  completion  of 
individual  projects  on  time  and  within  budget,  is  the  most  important  measure  of  success  for  any 
IT  investment.  How  a  specific  investment  contributes  to  mission  accomplishment  will  vary,  but 
will  typically  fall  into  the  categories  of  efficiency  or  effectiveness. 

•  Efficiency  benefits  result  from  improved  operations  and  are  the  benefits  typically 
identified  with  the  system,  such  as  cost  reductions  through  reduced  staffing,  lower 
overhead,  etc. 

•  Effectiveness  benefits  reflect  the  value  added  to  the  user  and  to  the  organization  or  the 
organization's  clients  (e.g.,  more  timely  response  to  inquiries).  These  benefits  are  service 
improvements  not  provided  by  the  status  quo. 

Cost  and  schedule  measures  are  highly  interrelated  with  benefit  measures  through  the  required 
linkage  of  dollars  or  time  to  the  achievement  of  a  particular  performance  result  is  achieved. 
Typically,  the  following  cost  and  schedule  data  are  measured  for  IT  investments: 
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1.  Return  on  Investment  (ROI). 

2.  Current  project  funding  support. 

3 .  V alue  of  benefits($  value) . 

4.  Projected  project  costs. 

5.  Rate  of  budget  expenditures  compared  to  projections. 

6.  Adherence  to  baseline  schedule/time-frame. 

The  Earned  Value  Management  approach  (section  8.11)  provides  an  effective  mechanism  for 
linking  cost,  schedule,  and  benefits  performance  for  acquisition  programs. 

5.3.1.4  Determine  Targets  and  Thresholds  For  Identified  Performance  Measure 

For  each  identified  outcome  performance  measure  the  program/project  manager  and  user  must 
define  a  target  and  a  threshold  value.  The  target  is  the  desired  outcome  or  value  for  the  level  of 
performance;  the  threshold  is  the  level  of  performance  below  which  the  investment  is  no  longer 
achieving  acceptable  results.  Targets  and  thresholds  can  be  developed  from  these  sources: 

•  Observations  or  studies  of  best  practices  (benchmarking). 

•  Values  mandated  in  the  requirements  documents. 

•  Historical  data  (if  available). 

•  Standards  and  requirements  imposed  by  higher  authority. 

Establishing  target  and  threshold  values  of  performance  measures  requires  analysis  of  existing 
data  or  the  initiation  of  data  collection  efforts. 

Primary  sources  of  target  and  threshold  values  are  the  requirement  documents,  user  strategic  and 
tactical  level  plans,  values  from  existing  system  performance  (if  the  investment  is  for  a  migration 
or  replacement  system,  and  DoD  policy  documents,  such  as  the  DoD  8000  series,  DoD  5000 
series,  MIL-STD-498. 

5.4  Validate  Feasibility  of  IT  Performance  Measures 

Once  an  IT  effort  baseline  is  developed,  each  performance  measure  must  be  evaluated  to  ensure 
that  its  collection,  verification,  and  validation  are  possible  and  that  collection  is  cost  effective. 
The  best  performance  measures  are  outputs  from  the  measured  process.  Measures  that  are 
natural  results  of  work  performed  are  less  expensive  and  more  accurate. 

Three  major  issues  must  be  addressed: 

1.  What  data  are  necessary  for  calculation  of  the  performance  measure,  when  are  the  data 
collected,  and  who  collects  the  data? 

2.  How  will  results  be  verified  and  validated? 

3.  What  is  the  cost  of  the  data  collection? 
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5.4.1  Identify  Data  Required  to  Calculate  the  Performance  Measure 

Frequently,  many  pieces  of  data  are  necessary  to  calculate  a  measure.  It  is  helpful  to  define  data 
components  to  clarify  the  required  data.  For  example,  at  a  minimum,  ROI  consists  of  both  a 
return  and  investment  component. 

Identify  all  data  necessary  to  calculate  the  performance  measurement,  when  the  data  will  be 
collected,  and  who  will  collect  the  data. 

5.4.2  Identify  the  Verification  and  Validation  Strategy  for  the  Data  Collection 

Verification  and  validation  of  measures  are  critical.  It  is  important  to  have  confidence  in  the 
measures  reported.  Identify  a  method  to  ensure  the  quality  of  the  information  that  is  reported. 

For  example,  developmental  and  project  testing  will  provide  insight  into  operation  after  the 
system  is  fielded.  The  testing  provides  the  opportunity  to  gather  and  analyze  preliminary  data. 

5.4.3  Identify  the  Collection  Cost 

Determine  the  administrative  burden  of  collecting  the  measure  (in  hours  or  dollars).  For 
example,  if  the  measure  is  captured  in  a  standard  generated  report,  the  costs  will  be  relatively 
low,  perhaps  a  half  hour  to  obtain  the  report  and  extract  the  necessary  information.  However,  a 
measure  that  requires  information  from  several  sources  and  subsequent  compilation  or  only 
exists  in  a  raw  form  may  have  higher  costs. 

5.4.4  Evaluate  Performance  Measurement  Feasibility 

The  answers  to  these  three  issues  determine  whether  the  performance  measures  can  be  used 
effectively.  If  any  measure  is  not  feasible,  it  must  be  excluded  from  the  performance  measures 
set.  Different  measures  or  a  different  collection  strategy  should  be  sought. 

5.5  Finalize  IT  Effort  Baseline  and  Performance  Measurement  Framework 

The  FT  Effort  baseline  should  contain  a  complete  set  of  feasible  performance  measures  that  are 
linked  to  the  IT  effort’s  mission  and  objectives.  The  baseline  must  now  be  finalized  by  obtaining 
stakeholders’  approval  and  by  establishing  the  framework  that  will  provide  the  needed  data  for 
progress  or  performance  evaluation.  Managers  must  ensure  that  measures  gauge  the  expected 
outcomes  from  the  stated  requirements. 

The  activities  involved  in  this  step  include  the  following: 

1.  Ensure  set  of  performance  measures  is  measuring  the  right  thing. 

2.  Ensure  set  of  performance  measures  has  the  right  measures. 

3.  Gain  consensus  for  the  performance  measurement  baseline. 

4.  Establish  data  collection  efforts  to  obtain  periodic  values  of  the  measures  in  the  baseline 

5.5.1  Ensure  Performance  Measures  Are  Measuring  the  Right  Things 

•  Does  the  set  of  measures  address  improvement  in  performance  of  objectives? 

1.  Are  all  objectives  covered  by  at  least  one  measure? 

2.  Does  set  of  measures  indicate  how  well  your  effort  achieves  those  objectives? 
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The  second  component  requires  evaluation  to  determine  whether  collecting  the  set  of 
performance  measures  will  indicate  continually  improving  performance. 

•  Does  the  set  of  measures  use  a  small  set  of  significant  performance  measures  that 
provide  a  clear  basis  for  assessing  accomplishment,  facilitate  decision  making,  and  focus 
on  accountability? 

Ensure  the  performance  measures  set  is  as  small  as  possible  to  gauge  the  accomplishment 
of  objectives,  support  any  directed  data  collection,  allow  management  to  make  informed 
choices,  and  focus  on  accountability.  It  is  better  to  have  few  measures  that  are  highly 
managed  than  a  large  set  of  measures  that  are  not  used. 

•  Do  the  measures  assess  the  “value-added”  contribution  made  by  the  IT  investment? 
Ensure  that  the  measures  can  capture  the  non-IT  benefits  of  IT  investments.  Examples  of 
these  types  of  measures  include  reductions  in  cost  to  perform  the  function,  increased 
satisfaction  levels  for  the  functional  area’s  customers,  or  decrease  in  the  lag  time  between 
requests  and  delivery  of  service. 

•  Do  the  measures  capture  the  requirements  of  internal  and  external  customers? 

•  Does  the  set  of  measures  address  the  internal  performance  of  the  IT  function  ? 

Ensure  that  the  set  of  measures  captures  the  effectiveness  and  efficiency  of  the  IT 
function  itself.  Examples  of  potential  measures  include  adherence  to  budgets  and 
schedules  or  technical  performance  criteria  such  as  response  time.  IT  investments  are 
intended  primarily  and  foremost  to  return  benefits  to  the  functions  that  utilize  IT.  Internal 
IT  measures  are  only  meaningful  if  the  IT  effort  has  external  benefits. 

•  Does  the  set  of  measures  address  the  benefits,  costs,  and  schedules? 

5.5.2  Ensure  Set  of  Performance  Measures  Has  the  Right  Measures 

•  Are  MNS  and  ORD  measures  linked  to  a  clear  outcome  (results  rather  than  inputs  or 
outputs)? 

Ensure  that  measures  gauge  accomplishment  of  outcomes,  results,  and  benefits  rather 
than  inputs  and  outputs.  Input  and  output  should  be  developed  later  by  the 
program/project  manager  and  the  user  to  define  test  and  acceptance  parameters. 

•  Is  the  set  of  measures  understood  at  all  levels  that  have  to  evaluate  and  use  the 
measures? 

Do  the  measures  support  effective  management  decisions?  Do  performance  measures 
communicate  achievements  to  internal  and  external  stakeholders? 

•  Is  the  set  of  measures  effective  in  prompting  action? 

It  is  better  to  measure  areas  that  are  "actionable,"  which  refers  to  areas  where 
performance  has  the  possibility  to  change  the  outcome. 
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•  Is  the  set  of  measures  accurate,  reliable,  valid,  verifiable,  and  cost  effective? 

Ensure  that  measures  produce  an  accurate,  reliable,  valid  and  verifiable  indication  of 
mission  success.  Make  sure  that  the  set  of  measures  is  built  on  data  that  are  available  at 
reasonable  cost,  appropriate,  and  timely  for  the  purpose.  Do  not  create  artificial  measures 
as  a  separate  reporting  item  that  are  not  otherwise  monitored  internally. 

•  Does  the  set  of  measures  include,  along  with  long-term  measures,  short-term  measures  or 
goals  that  show  interim  progress? 

Include  short-term  measures  or  goals  that  allow  for  demonstration  of  progress  and 
provide  a  performance-motivating  or  performance-sustaining  achievement  factor. 

5.5.3  Gain  Consensus  for  the  IT  Effort  Baseline  and  Measurement  Framework 

The  baseline  document  should  create  a  formal  agreement  among  the  requiring  activity,  the  IT 
manager  and  the  next  higher  level  of  oversight,  including  the  CIO  and  CFO  or  their  functional 
equivalents,  As  a  last  step,  it  is  useful  to  gain  consensus  from  all  stakeholders  that  the  baseline 
targets  will  satisfy  the  existing  needs.  Once  established,  the  baseline  should  be  placed  under 
configuration  management  to  ensure  that  all  stakeholders  agree  to  any  changes  to  the  baseline. 

For  each  IT  investment,  the  functional-level  manager  should  initiate  the  Investment 
Baseline/Performance  Agreement  (see  Appendix  A)  by  defining  the  following  data  elements: 

1.  The  mission  goals  of  the  functional  area  supported  by  the  proposed  investment. 

2.  The  current  operational  capability. 

3.  A  listing  of  the  enhanced  functional  capability  requirements. 

4.  For  each  requirement,  the  associated  expected  outcome  measures. 

5.  An  evaluation  of  how  closely  the  investment  supports  the  organization’s  strategic  goals. 

6.  A  definition  and  assessment  of  the  risks  and  benefits  associated  with  the  investment, 
including  those  incurred  if  the  project  is  not  done. 

7.  The  approach  for  conducting  a  post-deployment  assessment  of  the  operational  benefits 
and  capabilities  of  the  proposed  investment. 

8.  The  projected  cost  and  schedule  for  the  investment. 

Once  this  information  is  determined  at  the  functional  level,  and  the  investment  is  approved  for 
execution,  the  remainder  of  the  Investment  Baseline/Performance  Agreement  is  negotiated  with 
the  responsible  program/project-level  office  and  put  into  its  final  form.  Section  6  describes  this 
process. 

5.5.4  Determine  How  to  Collect,  Analyze,  Verify,  Validate  and  Track  Data 

A  final  consideration  is  to  determine  whether  mechanisms  are  in  place  to  generate  the  data  that 
are  needed  to  measure  progress  towards  goals  and  to  determine  whether  the  effort  is  within 
threshold  values. 
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6.  PERFORMANCE  MEASUREMENT  AT  THE  PROGRAM/PROJECT 
LEVEL 


This  section  describes  the  steps  necessary  to  develop  and  manage  the  Investment 
Baseline/Performance  Agreement  for  an  IT  program  or  project.  Baseline  development  is  used  to 
measure  progress  from  the  current  position  of  a  program,  project,  or  acquisition  toward  future 
goals.  Measurement  against  this  baseline  can  also  be  used  to  gauge  program  stability  and 
control. 

Section  3  identified  the  three  levels  (enterprise,  functional,  and  program/project)  interested  in 
IT  performance  measurement.  Now,  we  need  to  address  the  information  needed  to  provide  these 
groups  with  the  basis  to  develop  their  baseline,  measure  performance,  and  analyze  results. 

The  information  necessary  to  support  all  three  levels  of  interest  has  to  be  generated  and  captured 
during  the  development  and  implementation  of  the  IT  project.  The  key  information  will  link  the 
project’s  requirements  and  results  back  to  the  overall  goals  and  objectives  of  the  DoD. 

In  addition,  performance  measurement  will  capture  the  program/project-level  performance 
information  related  to  how  well  the  project  is  being  executed  —  is  it  on  time,  within  cost,  meeting 
milestones  and  requirements,  and  providing  the  anticipated  benefits  of  the  specific  project? 

What  is  the  key  information  needed  to  support  the  enterprise  and  functional  levels  of  the 
organization?  In  most  cases  this  information  is  different  from  what  would  be  needed  just  to  run  a 
project  and  consider  it  a  success.  This  information  provides  the  frame  of  reference  to  see  the 
results  of  the  project  on  a  larger  scale  -  how  does  it  help  improve  the  department’s  operations; 
how  do  the  results  compare  with  industry  best  practices;  how  does  it  support  the  overall 
approved  technical  architecture? 

To  answer  these  questions,  the  project  leader,  in  coordination  with  the  user,  has  to  ascertain  that 
the  essential  information  is  available.  This  information  includes: 

•  the  baseline  information  for  the  functional  area/activity  that  the  project  is  supposed  to 
“improve”  the  baseline  for  the  functional  area  is  termed  the  “external”  or  functional  baseline; 

•  the  best  practices  benchmark  (if  appropriate)  for  the  functional  area/activity  that  the  project 
is  supposed  to  improve; 

•  the  approved  project  targets  for  the  functional  area/activity,  which  are  the  basis  of  the 
project’s  existence  (what  it  should  deliver  in  terms  of  functional  results); 

•  the  timelines  for  achieving  the  functional  results;  and 

•  the  ROI  the  project  is  supposed  to  deliver  in  functional  improvement  (and  the  timeline  for 
achieving  the  ROI). 
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The  functional  baseline’s  benchmarks  and  project  targets  must  be  defined  in  measurable  terms. 
The  functional  area  must  be  defined  in  such  a  way  to  provide  the  key  cost,  time  (absolute  or  cycle 
time),  quality,  and  results/performance/customer  satisfaction  measurements.  These  quantified 
definitions  must  be  recorded  for  the  baseline  (start)  position,  the  benchmark  (best  practice) 
position,  and  the  project  target  position  (project  requirements  driver). 

The  project  manager  must  work  with  the  sponsoring  user  to  track  the  results  of  the  project  in  the 
same  quantifiable  functional  terms.  It  is  not  enough  that  a  project  is  executed  on  time,  within 
costs,  and  meeting  prescribed  project  requirements.  The  evaluation  of  the  project  must  also 
include  the  assessment  of  the  results  of  the  project  on  the  larger  functional  environment  and  how 
those  results  support  DoD  strategic  plans. 

There  are  four  major  steps  in  developing  project  performance  measures  (the  internal  baseline): 

1.  Identify  the  IT  project,  its  mission  and  objectives,  the  external,  functional  baseline,  the 
benchmark  (if  appropriate),  and  the  project  target  positions. 

2.  Define  baseline  performance  measures. 

3.  Validate  feasibility  of  performance  measures. 

4.  Finalize  performance  measurement  baseline  and  define  a  methodology  to  track  “external” 
project  results. 

It  is  impossible  to  overstate  the  need  to  incorporate  the  functional  user  in  every  step  of  this 
process  to  ensure  that  mission  accomplishment  remains  the  focus  of  the  project.  DoD  Regulation 
5000. 2-R  dictates  the  use  of  integrated  product  teams  (IPTs)  that  include  all  affected  functional 
areas  for  major  acquisition  programs;  the  approach  is  an  excellent  one  at  every  level. 

What  follows  is  a  roadmap  for  constructing  a  tailored  performance  measurement  baseline  for  an 
IT  project.  The  procedures  involve  answering  a  series  of  questions  and  recording  the  answers  on 
the  provided  worksheets.  Exhibit  6-1  presents  an  overview  of  the  major  steps  in  this  process. 

The  rest  of  this  section  provides  greater  detail  on  the  set  of  activities  within  the  process. 

A  performance  measurement  baseline  consists  of  two  major  components:  the  performance 
measure  itself,  and  the  threshold  and  target  values  for  the  performance  measures.  The  measures 
within  a  baseline  will  depend  upon  whether  the  baseline  is  project  (internal)  or  functional 
(external). 
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Source  Documentation 


STEP  1 


Worksheet  1: 
Project  Definition 


Exhibit  6-1:  Procedures 


6.1  Identify  IT  Project  and  its  Mission  and  Objectives 

Identifying  the  IT  effort  with  its  mission  and  objectives  and  the  external  functional  baseline  is  the 
most  critical  step  in  the  process  of  creating  an  internal  performance  measurement  baseline.  This 
step  is  critical  to  measuring  performance  based  on  the  ability  of  the  effort  to  successfully 
complete  its  mission  and  reach  its  objectives.  With  a  clear  understanding  of  why  the  effort  exists 
(mission)  and  must  accomplish  to  obtain  the  desired  results  (objectives),  and  the  external  frame 
of  reference  for  the  functional  area,  it  is  possible  to  assemble  a  set  of  performance  measures  that 
track  the  progress  of  the  project  and  functional  area. 
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Source  Documentation 

STEP  1 

Identify  IT  Effort  with 
Mission  and  Objectives, 
and  External  Baseline 

Worksheet  1: 

Project  Definition 

Worksheet  2: 

External  Environment 

1 

Go  to  STEP  2 

Exhibit  6-2:  Step  1 


This  step  requires  answering  the  following  questions: 

•  What  is  the  project?  (What  is  the  project  name;  Who  are  the  users  and  customers?) 

•  What  is  it  doing?  (What  kind  of  project  is  it  and  what  are  the  work  efforts?) 

•  Why  is  being  done?  (What  are  mission  and  objectives?) 

•  What  is  the  external  environment?  (What  are  the  functional  objectives?) 

Use  Worksheet  1  (Project  Definition)  in  Exhibit  6-3  for  recording  the  answers  to  the  first  three 
questions  and  Worksheet  2  (External  Environment)  in  Exhibit  6-6  for  answering  the  last 
question. 


Worksheet  1:  Project  Definition 

Project  Name:  _ 

Project  Leader:  _ 

Customers:  _ 

Project  Type:  _ 

Work  Efforts:  _ 

Mission:  _ 

Objectives:  Source: 

1. _  _ 

2. _  _ 

N._ _  _ 

Exhibit  6-3:  Worksheet  1  (Project  Definition) 
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6.1.1  Identify  the  Project 

Record  the  project  name  and  project  leader. 

Direct  project  customers  are  likely  to  be  the  offices  that  fund  the  effort.  They  also  have 
expectations  for  what  they  will  receive  in  return.  Indirect  customers  may  include  personnel  who 
are  affected  by  the  effort.  Examples  of  indirect  customers  are  users  of  a  new  system  or 
implementors  of  processes  and  procedures,  such  as  BPR  or  data  standardization.  Record  both 
customer  types  on  the  worksheet. 

When  thinking  of  customers,  project  managers  should  determine  what  their  expectations  are 
from  the  project  effort.  There  should  be  a  direct  relationship  between  their  expectations  and  the 
objectives  identified  in  later  steps. 

The  project  type  defines  the  effort  as  a  system  development,  a  migration  project,  or  other  type  of 
IT  investment. 

The  work  efforts  are  the  specific  tasks  to  be  accomplished  by  the  project. 


6.1.2  Identify  the  Mission 

The  purpose  of  any  IT  effort  should  translate  into  a  statement  of  mission  describing  benefits  of 
the  effort.  Consider  what  outcomes  are  expected  from  the  project.  The  contribution  to  the  DoD 
mission  should  be  clear. 

Record  answers  on  Worksheet  1  along  with  the  source  for  the  mission.  The  requiring  activity 
should  be  the  primary  source  for  the  mission  statement,  from  the  MNS,  ORD,  and  functional  area 
strategic  plan. 

The  following  example  illustrates  the  above: 


The  System  X  Project  is  developing  an  open-standards-based  system.  System  X  is 
in  the  middle  of  development.  The  Mission  Need  Statement  (MNS)  for  System  X 
provides  an  introductory  paragraph  which  states  that  System  X  “will  provide  the 
X  managers  the  ability  to  reduce  costs  by  Y%  while  reducing  processing  time  by  Z 
weeks ,  and  to  raise  customer  satisfaction  as  measured  by  numbers  of  repeat 
customers.  ”  In  addition,  System  X  will  be  based  on  open  standards.  It  will  use 
standardized  data.  A  Departmental  memorandum  also  states  that  to  succeed, 
System  X  must  interface  with  other  project  AISs  to  provide  timely  updates  of  an 
executive  information  database.  From  this  and  other  documented  information,  a 
mission  is  derived  declaring  that  “the  System  X  Program  will  deliver  an  open, 
standardized  system  providing  timely  and  concise  decision  information  to 
Departmental  decision  makers  through  the  extraction  of  project  data.  ” 

Exhibit  6-4:  Mission  Example 
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6.1.3  Identify  Objectives 

Next,  identify  the  objectives  of  the  effort.  Objectives  are  measurable  outcomes  that  are  critical 
to  the  accomplishment  of  the  IT  effort’s  mission.  The  objectives  are  the  primary  source  of  the 
measures  that  will  be  included  in  the  baseline. 

The  user  or  requiring  activity  should  provide  project  requirements,  statements  of  need  or 
effectiveness,  or  other  descriptions  of  what  the  effort  must  achieve  to  be  successful  in  the  initial 
Investment  Baseline/Performance  Agreement.  Developing  objectives  is  an  iterative  process  that 
distills  a  variety  of  information  into  a  set  of  measures  that  can  be  used  to  determine  status  and 
progress.  It  is  imperative  to  state  objectives  in  a  measurable  manner. 

After  reviewing  the  worksheet,  it  should  be  possible  to  construct  a  list  of  objectives  that  are 
critical  to  accomplishing  the  project’s  mission.  The  identification  of  customers  addressed  in  the 
first  question  and  their  expectations  from  the  project  should  translate  into  specific  objectives. 

Any  IT  effort  probably  has  multiple  customers.  It  is  likely  that  beyond  the  obvious  results,  there 
are  additional  performance  expectations  that  are  less  obvious. 

Specific  work  efforts  may  involve  contracted  efforts  that  have  SOWs  along  with  required 
deliverables.  The  SOWs  and  their  deliverables  should  be  mapped  into  the  objectives. 

Record  objectives  on  the  worksheet  as  they  are  formulated,  along  with  the  sources  of  information 
used.  These  sources  will  be  useful  in  developing  numerical  targets  and  threshold  values  for  the 
measures  that  are  based  on  the  objectives.  Ensure  that  project  objectives  on  Worksheet  1  align 
with  functional  area  objectives  on  Worksheet  2. 

The  following  example  illustrates  how  to  identify  IT  effort  objectives: 


The  results  expected  of  System  X  are  distributed  throughout  the  programmatic 
and  system  documentation.  The  MNS  and  Functional  Description  for  System  X 
directly  state  the  desired  results  from  the  system.  A  partial  list  of  the  desired 
results  from  System  X  includes: 

•  increase  in  the  population  served,  100,000  in  fiscal  year  1995,  by  30%  to  a 
population  of  130,000  by  the  end  of  fiscal  year  1998 

•  decrease  the  training  costs  of  system  users  from  $1,000  per  person  per  user  in 
fiscal  year  1995  by  25%  to  $750  per  user  by  end  of  fiscal  year  1998 

Exhibit  6-5:  Objectives  Example 

These  examples  of  desired  results  meet  the  criteria  for  quality  objectives.  Each  of  these  is 
results-oriented,  measurable,  and  supports  the  DoD  mission  with  regard  to  effective  and  efficient 
personnel  management. 

6.1.4  Identify  the  External  Environment 

The  information  required  to  complete  Worksheet  2  (External  Environment),  illustrated  in 
Exhibit  6-6,  should  be  provided  by  the  sponsoring  functional-level  organization  in  their  proposed 
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Investment  Baseline/Performance  Agreement.  Use  this  input  and  any  supporting  documentation 
to  clearly  identify  the  functional  objectives  that  the  project  supports.  Record  answers  on  the 
worksheet  along  with  the  name  of  the  director  and  title  of  the  sponsoring  office  or  organization. 


Worksheet  2:  External  Environment 

Functional  Area:  _ 

Project  Sponsor:  _ 

Functional  Baseline  Benchmark  Approved  Revised 

Objective:  Value  Value  Target  Estimate 

1.  _  _  _  _  _ 


2. 


3. 


Return  on  Investment: 

Functional  Economic  Analysis:  _  Date: 

Current  Estimate:  _  Date: 


Exhibit  6-6:  Worksheet  2  (External  Environment) 

6.2  Define  IT  Investment’s  Internal  Performance  Baseline 

Once  the  project  manager  has  identified  the  project  type  and  objectives  (the  critical  results  that 
must  occur  to  accomplish  your  mission),  performance  measures  are  selected  that  track  the 
accomplishment  of  your  objectives.  The  activities  involved  in  this  step  are  as  follows: 

1)  Identify  performance  measures. 

2)  Determine  performance  measures  targets  and  thresholds. 


Worksheet  3: 

Performance  Measures 

Internal 

W 

Worksheets  1  and  2 

- ► 

STEP  2 

Define  TT  Effort's 

Requirements 

- ► 

Source  Documentation 

T  1  D 

.. 

- ► 

internal  oas 

»eiine 

w 

Worksheet  4: 

Targets  and 

T 

Thresholds 

Go  to  STEP  3 

Exhibit  6-7:  Step  2 
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6.2.1  Identify  Performance  Measures 

Performance  measures  selected  for  the  baseline  should  be  critical  to  the  accomplishment  of  the 
effort.  Consider  the  specific  nature  of  the  project,  the  objectives  identified  in  Step  1,  and  all 
available  source  documentation.  Add  additional  performance  measures  to  Worksheet  3  that 
specifically  address  objectives  in  the  areas  of  benefits,  cost,  and  schedule. 

Worksheet  3:  Internal  Requirements 

Project  Type: 

Name:  Definition  and  Source: 

1.  ROIROI  of  project  taken  from  the  economic  analysis  for  the  Milestone  II  review. 

2.  Funding  Funding  in  support  contained  in  Exhibit  43. 

3.  Standards  Degree  of  compliance  with  prescribed  architecture(s). 

4.  Standard  Number  and  percentage  of  data  elements  in  data  model  that  are  standardized  data 
Data  elements  from  the  DoD  Data  Dictionary  System  (DDDS). 

5.  Benefits  Value  of  benefits  documented  in  the  economic  analysis  developed  for 
($  value)  Milestone  II  review. 

6.  Cost  Program  costs  documented  in  the  economic  analysis  developed  for  Milestone  II 

review. 

7.  Cycle  Reduction  in  administrative  lead  time  as  documented  in  the  benefits  portion 
Time  in  the  economic  analysis. 

8.  Quality  Accuracy,  response  time,  availability,  maintainability,  and  restoration.  Separate 

targets  and  thresholds  will  be  established  based  upon  the  TEMP,  and  data  will  be 
obtained  through  project  and  developmental  testing. 

9.  Customer  Ease  of  system  use.  Obtained  through  developmental  and  project  testing. 
Satisfaction 

10.  Budget  Measure  of  budget  execution,  and  rate  of  expenditures  compared  to 

projections  in  Exhibit  43. 

11.  Schedule  Measure  of  schedule/time-frame  adherence  to  baseline  established  at 

Milestone  II  review. 

(Custom) 

12 . N  Project-unique  measures  tailored  to  particular  requirements. 

Exhibit  6-8:  Worksheet  3 
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An  excellent  approach  to  performance  measurement  in  projects  requiring  software  development 
is  contained  in  Practical  Software  Measurement:  A  Guide  to  Objective  Program  Insight,  by  the 
Joint  Logistics  Commanders  Joint  Group  on  Systems  Engineering. 

6.2.1. 1  Benefit  Measures 

The  accomplishment  of  functional  missions  and  goals  (mission  benefit) ,  not  project  completion 
on  time  and  within  budget,  is  the  most  important  measure  of  success  for  any  IT  project.  How  a 
specific  project  contributes  to  mission  accomplishment  will  vary,  but  will  typically  fall  into  the 
categories  of  efficiency  or  effectiveness. 

Efficiency  benefits  result  from  improved  operations  and  are  the  benefits  typically  identified  with 
the  system,  such  as  cost  reductions  through  reduced  staffing,  lower  overhead,  etc. 

Effectiveness  benefits  reflect  the  value  added  to  the  user  and  to  the  organization  or  the 
organization's  clients  (e.g.,  more  timely  response  to  inquiries).  These  benefits  are  service 
improvements  not  provided  by  the  status  quo. 

The  manager  or  analyst  can  directly  measure  many  benefits  in  monetary  terms.  For  example, 
projects  for  modernization  or  replacement  of  existing  equipment  can  generate  operating  and 
support  savings  relative  to  the  status  quo.  This  benefit  is  quantifiable  in  direct  monetary  terms. 

Replacing  a  particular  work  step,  function,  or  piece  of  equipment  is  another  common  benefit. 

For  example,  administrative  lead  time  or  delay  can  be  reduced,  resulting  in  fewer  resources 
needed.  A  remote  job  entry  station  can  replace  the  central  data  entry  operation,  with  a  resulting 
cost  reduction.  Productivity  and  accuracy  gains  through  on-line  entry  may  also  translate  into 
personnel  savings. 

Benefits  that  are  not  specifically  monetary,  but  quantifiable,  can  often  be  converted  into 
equivalent  monetary  values.  These  benefits  include  labor  savings  and  error  reduction.  An 
efficiency/productivity  increase,  typically  expressed  in  person-years,  is  a  benefit  whose  value 
includes  all  direct  and  indirect  labor  costs.  Direct  labor  costs  are  salaries  or  hourly  wages,  while 
indirect  labor  costs  include  allowances,  leave,  and  fringe  benefits  to  reflect  the  full  cost  of 
providing  a  person-year  of  labor.  Documented  personnel  reductions  are  the  best  evidence  of 
monetary  benefit. 

An  important  category  of  effectiveness  benefits  deals  with  user  or  customer  satisfaction.  Exhibit 
6-9  contains  examples  of  potential  benefit  measures  in  this  category.  Many  effectiveness 
measures  however,  will  be  system-  or  function- specific  and  must  be  determined  on  a  case-by¬ 
case  basis  through  the  requirements  definition  process. 

A  useful  method  for  the  identification  of  additional  benefits  is  the  Delphi  technique,  in  which 
users,  managers,  and  professionals  with  knowledge  of  the  project  being  analyzed  form  a  group. 
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This  group  can  identify  possible  project  benefits.  Input  from  multiple  interested  parties  increases 
the  likelihood  that  the  functional  manager  will  include  all  important  benefits.  In  addition,  group 
analysis  aids  in  understanding  the  significance  or  insignificance  of  non-quantifiable  benefits. 


Measure  Name 

Definition 

Adaptability 

The  ability  of  a  product  to  become  suitable  for  a  specific  use  or 
situation  that  was  not  originally  intended 

Administrative 

Actions 

The  percent  of  or  cost  of  administrative  actions  required  (e.g., 
reduction  in  the  number  of  administrative  appeals) 

Communicability 

The  ability  of  a  product  or  service  to  be  reliably  transmitted  over  a 
medium  and  understood  by  the  receiver 

Compliance 

The  ability  to  meet  legislative  or  regulatory  mandates 

Flexibility 

The  ability  of  force  structure  to  adapt  to  surge  and  changing  mission 
requirements 

Morale 

The  measure  of  employee  attitude  to  work 

Price 

End  users’  satisfaction  with  what  they  are  paying  for  the  product  or 
services  as  compared  to  their  other  options 

Process  Time 

The  elapsed  time  between  the  commencement  and  completion  of  an 
activity 

Quality 

End  users’  perceived  quality  of  the  products  and  services  delivered  by 
the  program 

Response  Time 

The  elapsed  time  between  the  request  for  a  product  and  when  the 
system  or  component  begins  to  process  the  request 

Responsiveness 

End  users’  perception  that  they  get  what  they  need  when  they  need  it 

Re-usability 

Ability  of  the  system  or  parts  of  the  system  to  be  used  again 

Service  Life 

Length  of  time  the  equipment  will  be  able  to  support  the  operation 

Set-up  Time 

The  period  of  time  during  which  a  system  or  component  is  being 
prepared  for  a  specific  operation 

Simplicity 

An  evaluation  of  difficulty  of  performing  a  particular  operation 

Speed 

The  amount  of  time  necessary  to  respond  to  project  requirements 

Staffing 

Structure 

Changes  to  the  structure  of  the  project  unit 

System 

Procurement 

The  time  and  difficulty  of  procuring  the  system 

Turnaround 

Time 

The  elapsed  time  between  the  submission  of  a  request  for  a  product 
and  delivery  of  completed  product 

Understandabilit 

y 

The  clarity  of  the  system  and  its  functions 

Exhibit  6-9:  Examples  of  User/Customer  Satisfaction  Measures 
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Another  way  to  look  at  how  the  user  is  satisfied  is  to  set  up  a  perspective  view.  The  diagram 
below  shows  performance  measures  from  two  perspectives:  the  user  which  is  on  the  mission 
side  of  the  axis  and  the  service  provider  which  is  on  the  technology  side  of  the  axis. 


Performance  M  easures 
PERSPECTIVE 


Capability 
Simplicity 
Flexibility 
Serviceability 
Survivability 
Effectiveness 
Responsiveness 
Re-usability 
Cycle  Time 
Price 

Understandability 
Value/Quality 

Communicability 

Compliance 

Scalability 

Infrastructure 


Mission  (User) 


Technology 
(Service  Provider ) 


Quality 

Reliability 

Adaptability 

Timeliness/  Speed 

Efficiency 

Procureability 


Exhibit  6-10:  Another  View  -  User/Customer  Satisfaction  Measures 

Mission  accomplishment  is,  again,  the  most  important  measure  of  success  for  any  IT  project. 
How  IT  contributes  to  mission  accomplishment  tells  whether  an  investment  or  any  additional 
investment  should  be  made.  We  are  very  proficient  measuring  the  Technology  side  of  the 
equation,  but  proficiency  means  little  if  it  does  not  meet  the  user’s  perception  of  capability, 
understandability,  flexibility,  price  —  in  a  word,  Mission  ACCOMPLISHMENT,  Mission 
SUCCESS  -  now  will  typically  fall  into  the  categories  of  efficiency  or  effectiveness. 


6.2. 1.2  Cost  and  Schedule  Measures 

Cost  and  schedule  measures  are  highly  interrelated.  Common  to  both  is  the  required  linkage  of 
dollars  or  time  to  the  achievement  of  a  milestone.  A  milestone  is  an  event  at  which  time  a 
particular  performance  result  is  achieved.  For  example,  within  the  Major  Automated  Information 
Systems  Review  Council  (MAISRC)  process,  the  Concept  Studies  Decision,  Concept 
Demonstration  Decision,  the  Development  Decision,  the  Production  Decision,  and  Major 
Modification  Decision  are  all  milestones. 
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Phase  0 

Concept  Exploration 


Milestone  0 
Approval  to  Conducl 
Concept  Studies 

•  Verifies  Need 

•  Authorizes 
Concept 

Exploration  Phase 
^  Activities  and 
Resources 


Activities 


Assess  Technical  and 
Functional  Alternatives 

Appoint  PM 

Review  AIS 
Mission/Use 

Evaluate  Program 
Strategies 

Prioritize  Functional 
Requirements 

Assess  Risks 


Outcomes 


•  Selected  Program 
Concept 

•  AIS  Functional 
Description 

•  Selected  Program 
Strategy 

•  Established  Program 
Office 

•  Acquisition  Strategy 
and  Schedule 


Phase  I 

Program  Definition  and  Risk  Reduction 


Activities 


A 

i/1 


•  Demonstration  or 
Rapid  Prototype  to 
Validate  Program 
Concept 

* 

•  Testing 

•  Analysis  of 
Demonstrations  and 
Tests 


Outcomes 


•  Completed  Design 

-  Functional 
Requirements 

-  Standards 

-  Data  Elements 

•  Detailed 
Specifications 


Milestone  IIN 
''Approval  to  EnterN 
Engineering  and 
Manufacturing 
Development 
•  Validates  Design 
and  Specification 
Authorizes 
^Development  Phas^ 
ctivities  and 
Resources 


Milestone  I 
^Approval  to  Begin^ 
New  Acquisition 
Program 

•  Validates  Phase  0 
Outputs 

•  Authorizes 
Demonstration  and/ 
Validation  Phase/ 
\ctivities  and  . 


Phase  II 

Engineering/Manufacturing  Developmen 


Activities 


•  Develop  Full-Scale 
System  Prototypes 

•  Component,  System, 
and  Operational 
Testing 

•  Security  Testing 

•  Standards  and  Inter¬ 
operability  Testing 

•  Analysis  of  Test 
Results 


Outcomes 


•  Operational 
Prototype 

•  Production 
Specifications 

•  Complete  Logistics* 
Test,  and  Deployment 
Plans 


Phases  III  &  IV 

Production,  Deployment,  Operations  Support 

Activities 

Outcomes 

•  Operational 

Capability 

Contract 

•  Deploy  System 

•  Program  Benefits 

•  Assess  Operations 

Assessment 

•  Assess  Need  for 

•  Operational 

Modifications 

Assessment 

•  Mission  Validated 

•  Benefits  Collected 
and  Evaluated 

Exhibit  6-11:  Milestone  Reviews  and  Phases 

For  some  AIS  Development  activities,  this  level  of  milestones  might  be  at  too  high  a  level  of 
granularity.  The  phases  that  precede  each  of  the  milestones  can  be  broken  into  smaller 
performance  envelopes,  each  ending  with  a  milestone.  For  example,  the  performance  envelopes 
of  Phase  0,  Concept  Exploration  and  Definition,  are  (1)  Requirements  Definition,  (2)  Market 
Survey,  and  (3)  Risk  Assessment.  Each  of  these  performance  envelopes  ends  with  a  milestone. 
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You  can  break  down  these  performance  envelopes  further  until  you  obtain  milestones  at  the 
appropriate  granularity. 

The  IT  manager  needs  to  decide  the  granularity  at  which  to  track  the  progress  of  the  effort. 
Regardless  of  the  milestone  granularity,  it  is  imperative  that  costs  and  schedule  are  linked  to  the 
achievement  of  consistent  milestones. 

6.2. 1.2.1  Cost  Measures 

Cost  measures  gauge  the  number  of  investment  dollars  needed  to  achieve  a  particular  milestone. 
For  example,  a  performance  measure  can  be  the  dollars  necessary  to  move  an  AIS  Development 
effort  through  Phase  0.  Another  more  granular  measure  might  be  the  investment  required  to 
perform  the  Requirements  Definition  portion  of  Phase  0.  The  granularity  of  the  performance 
result  is  not  important;  what  is  important  is  tying  the  investment  of  dollars  to  the  achievement  of 
some  result.  This  task  is  decidedly  different  from  measuring  whether  budgeted  dollars  were 
spent  in  a  particular  fiscal  year. 

6.2.1.2.2  Schedule  Measures 

Schedule  measures  gauge  the  amount  of  time  necessary  to  obtain  a  particular  performance  result. 
Using  the  example  stated  above,  a  performance  measure  can  be  the  amount  of  time  necessary  to 
move  an  AIS  development  effort  through  Phase  0  or  the  amount  of  time  to  perform  the 
Requirements  Definition  portion  of  Phase  0.  What  is  important  is  that  your  schedule  measure  is 
tied  to  the  same  performance  result  as  your  cost  measure. 

Baselines  should  contain  major  events  that  have  impact  on  the  effort.  Achieving  these  events  on 
time  may  demonstrate  satisfactory  progress.  For  each  effort  you  can  establish  a  target  date  that  is 
based  upon  contractual  requirements  or  the  need  to  complete  an  event  before  another  can  start. 
Thresholds  for  these  events  can  be  set  by  policy  (90  days  beyond  target)  or  by  absolute  need 
when  there  is  no  slack  in  the  schedule.  Candidate  schedule  events  include: 

•  receipt  of  deliverables  required  by  contracts; 

•  initiation  of  testing  (developmental,  project,  follow-on); 

•  design  reviews  and  sign-offs; 

•  achievement  of  initial  or  full  project  capability; 

•  establishment  of  a  specified  level  of  performance; 

•  development  of  plans;  and 

•  completion  of  construction  or  installation. 

At  the  program  and  enterprise  levels,  the  number  of  schedule  breaches  per  year  may  indicate  the 
impact  of  oversight  and  other  policies.  While  a  count  of  breaches  in  one  year  is  not  a  meaningful 
measure,  trends  established  over  time  can  indicate  the  impact  of  new  policies.  Performance 
measure  users  at  all  levels  should  be  interested  in  determining  the  financial  impact  of  schedule 
breaches. 
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6.2.2  Determine  Targets  and  Thresholds  For  Identified  Performance  Measure 

Next,  for  each  identified  performance  measure  in  the  previous  two  activities,  determine  its 
threshold  and  objective.  The  target  is  the  desired  outcome  or  value  for  the  level  of  performance; 
it  is  what  you  want  to  achieve.  The  threshold  is  the  level  of  performance  below  which  the 
program,  project,  or  acquisition  is  no  longer  achieving  acceptable  results.  To  develop  targets,  you 
should  consider  these  questions: 

•  What  is  the  level  of  performance  observed  from  studies  of  best  practices  (benchmarking)? 

•  Are  there  standards/goals  for  this  measure? 

•  What  values  are  contained  in  the  program’s  background  documents? 

•  Are  historic  data  available  upon  which  to  base  the  measure? 

•  Are  data  accurate  and  reliable? 

•  Are  performance  targets  realistic? 

•  Do  performance  targets  represent  efficiency  and  effectiveness? 

•  Will  performance  targets  demonstrate  achievement  of  the  desired  result? 

•  How  can  we  identify  and  adopt  the  best  practices  to  improve  performance  (i.e., 
benchmarking)? 

•  When  does  an  event  need  to  take  place  to  complement  other  activities? 

•  How  does  performance  compare  to  the  best  practices  in  industry? 

Establishing  target  and  threshold  values  of  performance  measures  requires  you  to  analyze 
existing  data  or  initiate  data  collection  efforts.  Because  historic  data  necessary  to  develop  targets 
may  not  be  available,  you  may  only  need  to  estimate  rough,  reasonably  achievable  targets.  You 
may  refine  candidate  values  as  data  accumulates,  but  changes  must  be  agreed  upon  by  all  parties 
approving  the  baseline.  Enter  the  initial  values  for  the  target  and  threshold  values  in  the  Target 
and  Threshold  columns  of  Worksheet  4,  illustrated  in  Exhibit  6-12. 


Worksheet  4:  Target  and  Threshold  Values 

Number:  Measure  Target  Threshold 

1. _ _ 

2. _ _ 

3.  _ _ 

4.  _ _ 


N. 


Exhibit  6-12:  Worksheet  4  (Target  and  Threshold  Values) 

Primary  sources  of  target  and  threshold  values  are  the  requirements  documents,  user  strategic  and 
tactical  level  plans,  values  from  existing  system  performance  (for  migration  and  replacement 
systems),  and  DoD  policy  documents,  such  as  the  DoD  8000  series,  DoD  5000  series,  MIL-STD- 
498.  You  should  also  review  Worksheet  2,  which  describes  the  functional  area  objectives  and 
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measures.  An  alternative  source  of  performance  data  is  benchmarking.  Remember  to  repeat  this 
step  for  each  proposed  measure  until  you  have  completed  your  set  of  measures. 

6.3  Validate  Feasibility  of  Performance  Measures 

Once  your  preliminary  internal  baseline  is  developed,  Step  3  guides  you  through  evaluating  each 
measure  to  ensure  that  its  collection,  verification,  and  validation  are  possible  and  that  collection 
is  cost  effective.  The  best  performance  measures  are  outputs  from  the  measured  process. 
Measures  that  are  natural  results  of  work  performed  are  less  expensive  and  more  accurate. 
During  this  step,  when  you  discover  an  unfeasible  performance  measure,  you  will  refine  the 
measure,  return  to  Step  2  to  customize  a  new  measure,  or  delete  the  unfeasible  measure. 


Worksheets  3  and  4 

STEP  3 

V alidate  Feasibility 
of 

Performance  Measures 

Worksheet  5: 

Validation  Worksheet 

1 

Go  to  STEP  4 

Exhibit  6-13:  Step  3 


Using  Worksheet  3  as  an  input,  you  will  complete  a  copy  of  Worksheet  5  (Validation  Worksheet) 
for  each  performance  measure  (Refer  to  Exhibit  6-14).  Worksheet  5  evaluates  three  major 
issues: 

1.  What  data  are  necessary  for  calculation  of  the  performance  measure,  when  are  the  data 
collected,  and  who  collects  the  data? 

2.  How  to  verify  and  validate  the  results  to  ensure  that  results  are  accurate? 

3.  What  is  the  cost  of  the  data  collection? 

The  answers  to  these  questions  determine  whether  each  measure  is  cost  effective  to  collect. 

The  activities  of  this  step,  which  are  repeated  for  each  identified  performance  measure,  are  listed 
below: 

1.  Identify  data  required  to  calculate  the  performance  measure,  and  when  and  by  whom  the 
data  are  collected. 

2.  Identify  the  verification  and  validation  strategy  for  the  data  collection. 

3.  Identify  the  collection  cost. 

4.  Evaluate  whether  the  performance  measurement  is  feasible. 
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If  the  collection  of  this  performance  measure  is  not  cost  effective,  either  reconsider  means  of 
collecting  the  measure  or  return  to  Step  2  and  identify  a  different  performance  measure  that  can 
more  effectively  gauge  this  objective. 

Worksheet  5:  Validation  Worksheet 


Measure  #: _  Measure  Name: _ 


Data  Required  When  Collected  Who  Collects 


Verification/ 

Validation  Strategy: - 

Collection  Cost:  _ 

Cost-Effective 

Collection:  YES  /  NO 

Exhibit  6-14:  Worksheet  5  (Validation  Worksheet) 

6.3.1  Identify  Data  Required  to  Calculate  the  Performance  Measure 

Frequently,  many  pieces  of  data  are  necessary  to  calculate  a  measure.  It  is  helpful  to  define  data 
components  to  clarify  the  required  data.  For  example,  at  a  minimum,  ROI  consists  of  both  a 
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return  and  investment  component.  List  all  data  necessary  to  calculate  the  performance 
measurement,  when  the  data  will  be  collected,  and  who  will  collect  the  data. 

6.3.2  Identify  the  Verification  and  Validation  Strategy  for  the  Data  Collection 

Verification  and  validation  of  measures  are  critical.  It  is  important  that  you  have  confidence  in 
the  measures  that  you  report.  Identify  a  method  by  which  you  can  ensure  the  quality  of  the 
information  that  you  report.  For  example,  developmental  and  project  testing  will  provide  insight 
into  operation  after  the  system  is  fielded.  The  testing  provides  the  opportunity  to  gather  and 
analyze  preliminary  data. 

6.3.3  Identify  the  Collection  Cost 

Determine  the  administrative  burden  of  collecting  the  measure  (in  hours  or  dollars).  For 
example,  if  the  measure  is  captured  in  a  standard  generated  report,  the  costs  will  be  relatively 
low,  perhaps  a  half  hour  to  obtain  the  report  and  extract  the  necessary  information.  However,  a 
measure  that  requires  information  from  several  sources  and  subsequent  compilation  or  only 
exists  in  a  raw  form  may  have  higher  costs. 

6.3.4  Evaluate  Performance  Measurement  Feasibility 

Review  the  information  on  Worksheet  5  to  determine  whether  you  can  collect  the  performance 
measure  cost  effectively.  If  the  measure  is  not  feasible,  there  may  be  a  better  performance 
measure  or  a  different  collection  strategy  that  you  can  use.  Measures  that  are  not  feasible  should 
be  excluded  from  your  performance  measures  set. 

6.4  Finalize  Investment  Baseline/Performance  Agreement 

At  this  point  all  feasible  performance  measures  that  are  linked  to  your  IT  effort’s  mission  and 
objectives  have  been  identified.  The  previous  step  determined  that  you  could  collect,  verify,  and 
validate  each  of  your  performance  measures.  You  also  determined  that  the  collection, 
verification,  and  validation  was  cost  effective.  Step  4  finalizes  the  process  by  obtaining  the 
stakeholders’  approval  and  by  establishing  procedures  that  will  provide  the  needed  data  for 
progress  or  performance  evaluation. 


Worksheets  1  and  2 

Worksheet  6A: 

- ► 

STEP  4 

Quality  Checklist  ^ 

Worksheets  3  and  4 

-  ► 

Finalize  Performance 

Set  of  Worksheet  5 

- ► 

Measurement  Baseline 

Worksheet  6B: 

Objectives  Coveraqge 

Worksheet 

Exhibit  6-15:  Step  4 

Step  4  examines  the  set  of  performance  measures  as  a  whole.  Using  Worksheet  6A  (Exhibit  6- 
16)  as  a  checklist,  evaluate  the  set  of  performance  measures  to  determine  that  they  are  measuring 
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the  right  things  and  that  the  right  measures  are  being  used.  Worksheet  6B  (Exhibit  6-17)  helps 
ensure  that  all  objectives  outlined  in  Worksheet  1  are  measured  with  at  least  one  performance 
measure.  If  performance  measures  do  not  meet  the  criteria  discussed  in  this  step,  delete 
ineffective  performance  measures  or  return  to  Step  2.  Within  Step  2,  define  additional  measures 
that  meet  the  criteria  discussed  in  this  step.  In  some  rare  cases,  a  measure  may  be  unable  to  meet 
one  of  the  criteria  listed  below  because  of  the  nature  of  the  function,  the  cost  associated  with 
particular  collections,  or  other  reasons.  It  is  important  to  have  these  reasons  documented. 

The  activities  involved  in  this  step  include  the  following: 

1)  Ensure  set  of  performance  measures  is  measuring  the  right  thing. 

2)  Ensure  set  of  performance  measures  has  the  right  measures. 

3)  Gain  consensus  for  the  performance  measurement  baseline. 

4)  Establish  data  collection  efforts  to  obtain  periodic  values  of  the  measures  in  the 
baseline 


YES 

NO 

1.  Are  we  measuring  the  right  thing? 

•  Does  the  set  of  measures  address  improvement  in  performance  of 
objectives? 

•  Does  the  set  of  measures  use  a  small  set  of  significant  performance 

measures  that  provide  a  clear  basis  for  assessing  accomplishment,  facilitate 
decision  making,  and  focus  on  accountability? 

•  Does  the  set  of  measures  assess  the  “value-added”  contribution  made  by  the 
IT  investment? 

•  Does  the  set  of  measures  capture  the  requirements  of  internal  and  external 
customers? 

•  Does  the  set  of  measures  address  the  external  performance  of  the  functional 
area? 

•  Does  the  set  of  measures  address  the  benefits,  costs,  and  schedules? 

2.  Do  we  have  the  right  measures? 

•  Are  most  measures  linked  to  a  clear  outcome  (results  rather  than  inputs  or 
outputs)? 

•  Is  the  set  of  measures  understood  at  all  levels  that  have  to  evaluate  and  use 
the  measures? 

•  Is  the  set  of  measures  effective  in  prompting  action? 

•  Is  the  set  of  measures  accurate,  reliable,  valid,  verifiable,  and 
cost  effective? 

•  Does  the  set  of  measures  include,  along  with  long-term  measures,  short-term 
measures  or  goals  that  show  interim  progress? 

Exhibit  6-16:  Worksheet  6 A  (Quality  Checklist) 
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Performance 

Measure 

Objective 

1 

Objective  2 

Objective  3 

... 

Objective  n 

Performance 
Measure  1 

Performance 
Measure  2 

Performance 
Measure  3 

• 

• 

• 

Performance 
Measure  m 

Exhibit  6-17:  Worksheet  6B  (Objectives  Coverage  Worksheet) 


6.4.1  Ensure  Set  of  Performance  Measures  is  Measuring  the  Right  Thing 

When  any  of  the  below  listed  criteria  are  not  met,  the  project  manager  should  consider  whether  a 
performance  measure  should  be  added  or  altered.  Using  Worksheet  6A,  evaluate  the  set  of 
performance  measures  against  the  following  questions: 

•  Are  all  objectives  covered  by  at  least  one  measure? 

Complete  Worksheet  6B  to  answer.  Worksheet  6B  is  a  matrix  with  objectives  in  a 
column  and  each  performance  measure  in  a  row.  Place  an  “X”  in  the  intersection  of  the 
performance  measure  and  objective  when  the  performance  measure  gauges  the 
accomplishment  of  the  objective.  After  this  has  been  done  for  each  performance 
measure,  check  to  ensure  that  all  objectives  are  covered  by  at  least  one  measure. 

•  Does  the  set  of  measures  indicate  how  well  the  effort  achieves  those  objectives? 

This  requires  an  evaluation  to  determine  whether  collecting  the  set  of  performance 
measures  will  indicate  continually  improving  performance. 

•  Does  the  set  of  measures  use  a  small  set  of  significant  performance  measures  that 
provide  a  clear  basis  for  assessing  accomplishment,  facilitate  decision  making,  and  focus 
on  accountability? 

Ensure  that  the  set  of  performance  measures  is  as  small  as  possible  to  gauge  the 
accomplishment  of  objectives,  provide  essential  information,  allow  management  to  make 
informed  choices,  and  focus  on  accountability.  It  is  better  to  have  few  measures  that  are 
highly  managed  than  a  large  set  of  measures  that  are  not  used. 

•  Does  the  set  of  measures  assess  the  “value-added”  contribution  made  by  the  IT 
investment? 

Ensure  the  set  of  measures  captures  the  non-IT  benefits  of  the  investment.  Examine  the 
set  of  measures  and  make  sure  they  capture  the  functional  benefit  that  the  effort  provides. 
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Examples  of  these  types  of  measures  include  reductions  in  cost  to  perform  the  function, 
increased  satisfaction  levels  for  the  functional  area’s  customers,  or  decrease  in  the  lag 
time  between  requests  and  delivery  of  service. 

•  Does  the  set  of  measures  capture  the  requirements  of  internal  and  external  customers? 
Ensure  that  your  set  of  measures  includes  measures  of  customer  satisfaction. 

•  Does  the  set  of  measures  address  the  internal  performance  of  the  IT  function  ? 

Ensure  the  set  of  measures  captures  the  effectiveness  and  efficiency  of  the  IT  function 
itself.  Examples  of  potential  measures  include  adherence  to  budgets  and  schedules  or 
technical  performance  criteria  such  as  response  time.  IT  investments  are  intended 
primarily  and  foremost  to  return  benefits  to  the  functions  that  utilize  IT.  Internal  IT 
measures  are  only  meaningful  if  the  IT  effort  has  external  benefits. 

•  Does  the  set  of  measures  address  the  benefits,  costs,  and  schedules? 

Assess  whether  the  set  of  performance  measures  includes  benefit,  cost,  and  schedule 
measures. 

6.4.2  Ensure  Set  of  Performance  Measures  Has  the  Right  Measures 

When  any  of  the  below  listed  criteria  are  not  met,  consider  whether  to  alter  an  existing 
performance  or  add  a  new  measure.  This  activity  consists  of  evaluating  the  set  of  performance 
measures  against  the  following  questions: 

•  Are  most  measures  linked  to  a  clear  outcome  (results  rather  than  inputs  or  outputs )? 
Ensure  that  measures  gauge  accomplishment  of  outcomes,  results,  and  benefits  rather 
than  inputs  and  outputs.  Inputs  and  outputs  only  have  meaning  if  outputs  result  in  some 
desirable  end. 

•  Is  the  set  of  measures  understood  at  all  levels  that  have  to  evaluate  and  use  the 
measures? 

Ensure  the  definitions  of  all  performance  measures  will  be  understood  at  all  levels  of 
review.  Are  they  the  right  measures  from  which  to  make  effective  management 
decisions?  Can  performance  measures  communicate  project  achievements  to  internal  and 
external  stakeholders? 

•  Is  the  set  of  measures  effective  in  prompting  action? 

Ensure  the  set  of  performance  measures  is  effective  for  managing  your  IT  effort.  It  is 
better  to  measure  areas  that  are  "actionable,"  which  refers  to  areas  where  performance  has 
the  possibility  to  change  the  outcome.  For  example,  customers  served  as  a  performance 
measure  depends  on  customer  arrivals.  Better  measures  of  the  organization's 
performance  in  customer  service  may  be  average  wait  time  per  customer  or  case 
resolution  lapse  time. 
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•  Is  the  set  of  measures  accurate,  reliable,  valid,  verifiable,  and  cost  effective? 

Ensure  the  set  of  measures  will  produce  an  accurate,  reliable,  valid  and  verifiable 
indication  of  mission  accomplishment.  Reexamine  measures  from  Worksheet  3  to 
ensure  that  they  are  effective  and  efficient.  Make  sure  that  the  set  of  measures  is  built  on 
data  that  are  available  at  reasonable  cost,  appropriate,  and  timely  for  the  purpose.  The 
best  measures  are  tightly  integrated  into  the  business  process.  Do  not  create  artificial 
measures  as  a  separate  reporting  item  that  are  not  otherwise  monitored  internally. 
Measures  that  are  the  natural  result  of  functional  processes  are  most  likely  to  be 
successfully  collected. 

•  Does  the  set  of  measures  include,  along  with  long-term  measures,  short-term  measures  or 
goals  that  show  interim  progress? 

Ensure  your  set  of  measures  includes  short-term  measures.  Measures  that  demonstrate 
benefit  during  a  given  period  of  performance  are  preferable  to  longer-term  measures. 
Include,  along  with  any  long-term  measures,  short-term  measures  or  goals  that  allow  for 
demonstration  of  progress  and  provide  a  performance-motivating  or  performance- 
sustaining  achievement  factor. 

6.4.3  Gain  Consensus  for  the  Performance  Measurement  Baseline 

The  baseline  document  should  create  a  formal  agreement  between  the  IT  manager  and  the  next 
higher  level  of  oversight,  including  the  CIO  and  CFO  or  their  functional  equivalents.  As  a  last 
step,  it  is  useful  to  seek  comment  and  to  gain  consensus  from  all  stakeholders  that  meeting  the 
targets  within  the  baseline  will  satisfy  the  existing  needs.  Based  on  the  results,  it  might  be 
necessary  to  revisit  some  of  the  previous  steps  to  refine  the  set  of  performance  measures.  Once 
established,  the  baseline  should  be  controlled  using  the  techniques  of  configuration  management 
to  ensure  that  all  stakeholders  agree  to  any  changes  to  the  baseline. 

Appendix  A  contains  the  format  for  an  Investment  Baseline/Performance  Agreement  designed  to 
serve  as  the  formal  baseline  agreement  between  the  project  manager,  the  user  representative,  and 
the  financial  activity  providing  funding  for  the  effort. 

6.4.4  Establish  Data  Collection  Efforts  to  Obtain  Values  of  the  Measures  in  the  Baseline 

A  final  consideration  is  to  determine  whether  mechanisms  are  in  place  to  generate  the  data  that 
are  needed  to  measure  progress  towards  goals  and  to  determine  whether  the  effort  is  within 
threshold  values. 


6-21 


7.  CASE  STUDY  OF  A  HYPOTHETICAL  IT  PROJECT 


7.1  Introduction 

This  case  study  illustrates  the  use  of  performance  measures  in  describing  the  status  of  an  ongoing 
program,  the  Hypothetical  Information  Resource  Management  System  (HIRMS).  The  case  study 
follows  the  procedures  in  Sections  4-6  for  creating  an  internal  program  baseline.  The 
performance  measures  in  the  baseline  are  drawn  in  part  from  the  minimum  essential  set  of 
measures  contained  in  Section  6. 


This  case  study  is  fictional  and  has  been  developed  only  to  illustrate  the  procedures  that 
have  been  discussed  in  this  guide. 


7.2  Development  of  the  Baseline  for  the  HIRMS 

The  following  steps  are  defined  in  Section  6  of  this  guide,  and  their  application  will  be  illustrated 
in  the  subsections  that  follow. 

1.  Identify  the  IM/IT  effort  and  its  mission  and  objectives,  and  the  quantified  external 
functional  baseline,  benchmark,  if  appropriate,  and  project  target  positions. 

2.  Define  minimum  essential  performance  measures  and  additional  (custom)  performance 
measures  for  the  internal  project  performance  baseline. 

3.  Validate  feasibility  of  performance  measures. 

4.  Finalize  performance  measurement  baseline  and  define  methodology  to  track  external 
project  results. 

7.2.1  Step  1:  Identify  IT  Effort  and  its  Mission  and  Objectives,  and  the  External 
Functional  Baseline 


Source  Documentation 

STEP  1 

Identify  IT  Effort  with 
Mission  and  Objectives, 
and  External  Baseline 

Worksheet  1: 

Project  Definition 

Worksheet  2: 

External  Environment 

1 

Go  to  STEP  2 

Exhibit  7-1:  Step  1 
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The  first  step  is  to  identify  the  IM/IT  effort  and  its  mission  and  objectives.  The  mission  is  the 
reason  that  the  effort  exists.  Objectives  are  measurable  results,  critical  to  the  accomplishment  of 
the  mission.  Together,  the  mission  and  objectives  define  the  contribution  that  the  project  will 
make  to  the  achievement  of  the  functional  area  objectives  contained  in  the  external  baseline. 

The  activities  included  in  this  step  involve  answering  the  following  questions: 

•  What  is  the  project?  (Determine  project  name,  leader,  and  customers.) 

•  What  is  it  doing?  (Determine  the  project  type  and  work  efforts) 

•  Why  is  it  being  done?  (Determine  mission  and  objectives.) 

•  What  is  the  external  environment?  (Determine  functional  objectives.) 

Use  Worksheet  1  (Project  Definition),  illustrated  in  Exhibit  7-2,  for  recording  the  answers  to  the 
first  three  questions.  Worksheet  2  (External  Environment)  will  be  used  to  record  the  answer  to 
the  last  question. 

Activity  1:  Identify  the  project. 

The  initial  activity  involves  determining  the  name  of  the  project  and  the  name  of  the  project 
leader.  The  determination  of  the  customers  is  based  upon  identifying  the  office  providing  the 
funding  for  the  program  and  the  ultimate  users  of  the  system.  This  information  will  aid  in 
identifying  objectives. 

A  review  of  the  source  documentation  for  the  HIRMS  program  identifies  the  office  that  must 
obtain  the  funding  for  the  program  by  examining  the  program’s  Exhibit  43.  For  this  case  study 
this  is  the  Assistant  Secretary  of  Defense  for  Information  Resource  Management  (ASD(IRM)). 
This  is  the  primary  customer.  Additional  customers  are  the  users  of  the  system  throughout  the 
Department. 

Activity  2:  Identify  what  it  is  doing. 

The  second  question  involves  identifying  the  project  type  and  the  work  efforts  that  make  up  the 
program. 

For  the  purpose  of  this  case  study,  the  government  is  developing  and  implementing  an  HIRMS 
for  obtaining  and  managing  IT  equipment  and  services.  The  program  involves  the  development 
of  software  for  an  open  systems  environment  and  shared  databases  that  will  employ  standard  data 
elements.  The  project  type  is  therefore  an  automated  information  system  (AIS)  development. 
The  requirements  have  been  generated  following  several  BPR  studies  that  have  identified  ways 
to  incorporate  IT  to  reduce  the  costs  of  current  operations  while  improving  the  timeliness  and 
accuracy  of  information  provided  to  managers.  The  system  will  be  deployed  to  numerous  sites 
across  the  United  States. 

The  entries  for  work  efforts  are  from  the  highest  level  of  the  program’s  work  breakdown 
structure  (WBS).  More  specific  detail  is  contained  in  the  lower  levels  of  the  WBS  and  would  be 
more  useful  in  identifying  specific  program  objectives.  The  primary  work  efforts  include: 
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1)  development  of  an  automated  information  system, 

2)  deployment  of  the  system  to  sites  across  the  country  and  abroad,  and 

3)  training  of  users  and  maintainers  of  the  system. 

At  this  point  we  will  make  initial  entries  into  the  first  worksheet,  shown  in  Exhibit  7-2. 


Worksheet  1:  Project  Definition 

Project  Name:  Hypothetical  Information  Resource  Management  System 
Project  Leader:  I.  M.  Incharge 

Customers:  ASD(Information  Resource  Management),  system  users,  and  maintainers. 
Project  Type:  Automated  Information  System  Development 
Work  Efforts: 

1.  Develop  an  automated  information  system. 

2.  Deploy  the  system  to  sites  across  the  country  and  abroad. 

3.  Train  users  and  maintainers  of  the  system. 

Mission:  (Include  sources) _ 


Objectives:  Source: 

1. _  _ 

2. _  _ 

3. _  _ 


4.. 


5, 


6, 


Exhibit  7-2:  Worksheet  1 
Activity  3:  Identify  why  it  is  being  done. 

Why  is  this  project  needed?  The  answer  lies  within  the  mission  and  objectives  for  the  program. 
A  review  of  the  requirements  documentation  will  provide  the  answers.  The  source  documents 
may  use  terminology  other  than  mission  and  objectives.  The  project  manager  must  review  the 
source  documents  and  distill  their  contents  into  consistent  statements  of  mission  and  objectives. 
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This  iterative  process  attempts  to  remove  inconsistencies  and  focus  on  stating  the  mission  and 
objectives  in  such  a  manner  that  quantitative  measures  can  subsequently  be  developed 

Exhibit  7-3  illustrates  the  relationships  between  some  of  the  source  documents  that  provide  data 
to  aid  in  defining  the  program’s  mission  and  objectives  and  the  program’s  list  of  candidate 
performance  measures.  The  Program  Manager’s  Charter  (PMC),  Mission  Need  Statement 
(MNS),  functional  requirements,  and  the  Test  and  Evaluation  Master  Plan  (TEMP)  provide 
insight  into  program  mission  and  objectives,  and  the  expected  benefits  that  will  accrue.  The 
TEMP  specifically  identifies  levels  of  performance  that  the  system  must  demonstrate  to  be 
acceptable,  thus  allowing  product  and  deployment.  The  TEMP  is  a  primary  source  of 
performance  measurement  requirements. 


Exhibit  7-3:  Relationships  Between  Source  Documents  and  Performance  Measures 


Review  of  these  documents  should  provide  answers  to  the  questions  of  why  DoD  requires  the 
HIRMS.  The  answers  provide  the  mission  and  objectives  and  the  starting  point  for  identifying 
relevant  performance  measures  for  the  program. 

7.2.1. 1  Program  Manager’s  Charter  (PMC) 

If  a  PMC  is  developed,  it  serves  as  a  written  contract  between  the  program  manager  (PM)  and  the 
chartering  authority.  The  PMC: 

•  provides  the  authority  for  ensuring  that  AIS  development  and  project  transition  are 
conducted  within  a  clearly  established  management  framework; 

•  establishes  the  objectives,  scope,  organization,  responsibilities,  methods  of  operation,  and 
required  resources  for  the  AIS;  and 
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•  identifies  the  lines  of  authority  and  accountability,  such  as  relationships  among  the  OSD 
PSA,  heads  of  the  DoD  Components,  participating  and  supporting  organizations,  and  the 
AIS  PM. 

According  to  the  PMC,  the  purpose  of  HIRMS  is  to  establish  a  fully  functional  AIS,  which  will 
standardize  data  elements  and  support  uniform  business  practices  throughout  the  Department. 

The  scope  of  HIRMS  includes  all  contracting,  receipt,  storage,  and  distribution  activities  for 
goods  and  services  required  by  the  Department.  HIRMS  will  use  open  systems  and  relational 
database  technology  to  provide  timely  and  accurate  information  to  improve  the  management  of 
supplies  and  services. 

The  functional  objectives  of  the  HIRMS  are  listed  below. 

•  Support  the  use  of  standard  department  management  policies,  processes,  and  shareable 
data. 

•  Improve  timeliness,  accuracy,  and  effectiveness  of  management  information. 

•  Optimize,  streamline,  and  integrate  disparate  IRM  automated  systems,  subsystems,  and 
databases. 

•  Facilitate  the  department-wide  integration  of  a  standard,  robust,  management 
environment  through  the  implementation  of  standard  processes,  and  standard  shared  data. 

•  Provide  for  improved  data  management  and  data  integrity  by  electronic  input  of  selected 
data  to  a  logically  shared  data  repository.  Standard  data  and  data  transmissions  must  be 
employed.  The  capability  to  exchange  data  within  the  department,  other  government 
agencies,  and  with  industry  must  be  provided. 

•  Provide  information  exchange  capabilities  among  department  components  and  related 
functional  areas. 

•  Provide  for  use  of  department-wide  electronic  commerce/electronic  data  interchange 
(EC/EDI). 

•  Streamline  manual  management  processes,  including  the  automation  of  manual 
management  activities  and  the  ability  to  input  data  only  once  at  the  source. 

•  Provide  an  on-line  means  for  capturing  and  evaluating  customer  feedback  information. 

•  Provide  the  status  of  materials  that  are  on  order  or  on  hand  in  a  near-real-time 
environment  to  enable  department  managers  to  more  closely  monitor  the  assets  of  the 
department. 


7.2.1.2  Program  Mission  Need  Statement  (MNS) 

The  MNS  defines  and  documents  a  mission  need,  and  justifies  resource  expenditures  for  the 
identification  and  exploration  of  solutions  to  satisfy  the  need.  The  MNS  provides  the  basis  to 
ensure  that  the  system  developed  satisfies  the  requirements  as  stated  in  the  MNS. 

The  primary  mission  for  HIRMS  is  enhanced  customer  service  through  process  improvements, 
elimination  of  paperwork,  and  improved  automated  tools  for  system  users.  The  HIRMS  will 
incorporate  improved  and  standardized  business  practices  and  electronic  commerce  techniques. 
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These  actions  will  improve  customer  service  by  reducing  costly,  time-consuming  paperwork; 
facilitating  responsiveness  to  customer  inquiries;  and  facilitating  prompt  and  accurate  responses 
to  requests  for  information. 

The  business  process  improvements  that  will  be  introduced  by  the  HIRMS  will  facilitate 
successful  performance  of  management  functions  under  anticipated  budgetary  and  personnel 
constraints. 

The  HIRMS  will  provide  improved  ability  to  support  departmental  needs  by  more  efficiently  and 
effectively  providing  timely  response  to  managerial  requirements,  improved  visibility  of  assets, 
and  more  accurate  information  through  shared  data.  Expected  benefits  include: 

•  decreased  administrative  lead  time  (the  time  required  for  management  actions  should  be 
benchmarked  against  civilian  practices); 

•  elimination  of  labor  intensive  processes,  duplicate  data  entry,  and  paper-handling  tasks, 
enabling  managers  to  focus  on  functional  tasks  requiring  judgment  and  experience; 

•  increased  accuracy  on  pending  action  status  requests; 

•  improved  security  for  IRM  data; 

•  increased  customer  satisfaction; 

•  decreased  training  requirements  and  costs  due  to  the  standard  user  interface  and 
commonality  of  the  AIS  among  all  users; 

•  increased  readiness  from  improved  availability  of  supplies;  and 

•  enhanced  capture  of  up-to-date  accurate  information  resulting  in  more  efficient 
management  of  contracts,  which  leads  to  decreased  penalties  and  interest  for  late 
government  payment. 

7.2.1.3  Operational  Requirements  Document  (ORD) 

The  functional  requirements  developed  during  the  initial  phase  of  a  program  serve  as  the  basis 
for  the  subsequent  Operational  Requirements  Document  (ORD).  Both  of  these  documents  must 
contain  sufficiently  precise  definitions  of  the  requirements  so  that  potential  system  developers 
can  estimate  the  level  of  effort  required  for  development  and  deployment. 

The  following  are  excerpts  from  the  ORD  for  the  HIRMS. 

•  System  Response  Time:  The  response  time  between  the  user  initiating  any  command  and 
the  response  time  arriving  at  the  user’s  workstation  shall  be  three  seconds  maximum  with 
a  desired  interval  of  two  seconds,  regardless  of  the  number  of  concurrent  users. 

•  System  Availability:  The  local  site’s  system  shall  be  available  98%  of  the  time.  This  will 
not  include  hardware  outages  for  scheduled  system  backup  and  maintenance  time. 

7.2.1.4  Test  and  Evaluation  Master  Plan  (TEMP) 

A  major  component  of  a  TEMP  is  the  list  of  the  Minimal  Acceptable  Project  Performance 
Requirements  (MAOPRs)  for  the  program.  This  list  must  be  individualized  for  each  program 
and  relate  the  objectives  for  that  program  to  measures  that  can  be  obtained  through  testing  to  aid 
in  system  evaluation. 
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The  MAOPRs  represent  the  minimum  acceptable  project  effectiveness  and  project  suitability 
characteristics  and  performance  thresholds  against  which  the  system  will  be  evaluated.  Exhibit 
7-4  contains  the  MAOPRs  that  are  derived  from  the  HIRMS  MNS  and  functional  requirements. 


# 

MAOPR 

Value 

Linkage 

1 

Decreased  administrative  lead  time. 

95%  of  management  actions 
completed  within  96  hours. 

Mission  Need 
Statement 

2 

Decreased  cost  of  management  due  to 
streamlining  of  processes. 

75%  decrease  in  person-hours 
required  to  process  a  request. 

Mission  Need 
Statement 

3 

Provision  of  accurate  information  on 
requests  for  status  of  pending  actions. 

99%  accuracy. 

Mission  Need 
Statement 

El 

Protection  of  information  from 
unauthorized  access. 

Rejects  unauthorized  access  and 
intrusion  100%  of  the  time. 

Mission  Need 
Statement 

5 

User-friendly  design. 

At  least  75%  of  HIRMS  users 
express  satisfaction. 

Mission  Need 
Statement 

6 

Effective  training  program  for 
personnel  who  operate/maintain 
HIRMS. 

At  least  90%  of  trained  users 
achieve  certification. 

Mission  Need 
Statement 

7 

Sustainability  under  the  current 
personnel. 

No  increase  in  the  number  of 
personnel  or  the  entrance 
abilities  of  user  personnel. 

Mission  Need 
Statement 

8 

Acceptable  response  time  to  user 
commands. 

Response  time  after  initiating  a 
command  shall  be  3  seconds 
maximum  with  a  desired 
interval  of  2  seconds. 

Functional 

Requirements 

9 

Maximum  availability  to  sustain 
operations. 

Available  98%  of  time  for 
software  and  interfaces. 
Minimum  allowable  is  95%. 

Functional 

Requirements 

10 

Short  mean  time  to  restore  function. 

99%  of  system  faults  shall  be 
corrected  within  48  hours. 

Functional 

Requirements 

11 

Provision  of  the  capability  to  rebuild 
and  restore  databases. 

100%  of  data  from  last  backup 
is  recovered 

Functional 

Requirements 

Exhibit  7-4:  Minimal  Acceptable  Project  Performance  Requirements  for  the  HIRMS 

Once  the  mission  of  the  IRM  effort  has  been  identified  and  entered  into  the  worksheet,  identify 
the  program’s  objectives.  Objectives  are  measurable  outcomes  that  are  critical  to  the 
accomplishment  of  the  IRM  effort’s  mission. 

The  statement  of  mission  entered  into  the  worksheet  is  taken  from  the  MNS.  The  objectives 
listed  are  based  upon  the  project  performance  requirements  from  the  TEMP.  For  this  example, 
the  TEMP,  which  is  the  best  source  of  information  on  quantifiable  objectives,  is  the  synthesis  of 
material  from  other  source  documents.  The  completed  Worksheet  1  is  displayed  in  Exhibit  7-5. 
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Worksheet  1:  Project  Definition 

Project  Name:  Hypothetical  Information  Resource  Management  System 

Project  Leader:  I.  M.  Incharge 

Customers:  ASD(IRM),  system  users  and  maintainers. 

Project  Type:  Automated  Information  System  Development 

Work  Efforts:  1.  Develop  an  automated  information  system. 

2.  Deploy  the  system  to  sites  across  the  country  and  abroad. 

3.  Train  users  and  maintainers  of  the  system. 

Mission:  Enhanced  customer  service  through  process  improvements,  elimination  of  paperwork, 
and  improved  automated  tools  for  system  users.  The  HIRMS  will  incorporate  improved  and 
standardized  business  practices  and  electronic  commerce  techniques.  These  actions  will  improve 
customer  service  by  reducing  costly,  time-consuming  paperwork;  facilitating  responsiveness  to 
customer  inquiries;  and  facilitating  prompt  and  accurate  responses  to  requests  for  information 

Objectives: 

1.  Decreased  administrative  lead  time. 

Source: 

Mission  Need  Statement 

2.  Decreased  labor  cost  by  elimination  of  processes. 

Mission  Need  Statement 

3.  Provision  of  accurate  status  reports  on  pending  actions. 

Mission  Need  Statement 

4.  Protection  of  information  from  unauthorized  access. 

Mission  Need  Statement 

5.  User-friendly  design. 

Mission  Need  Statement 

6.  Effective  training  program  for  personnel  who  operate 
and/or  maintain  HIRMS. 

Mission  Need  Statement 

7.  Sustainability  under  the  current  personnel. 

Mission  Need  Statement 

8.  Acceptable  response  time  to  user  commands. 

Mission  Need  Statement 

9.  Maximum  availability  to  sustain  operations. 

Functional  Requirements 

10.  Short  mean  time  to  restore  function. 

Functional  Requirements 

11.  Capability  to  rebuild  and  restore  databases. 

Functional  Requirements 

Exhibit  7-5:  Completed  Worksheet  1 
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Activity  4:  Identify  the  external  environment. 


First,  determine  the  functional  area  to  which  the  effort  belongs.  DoD  Instruction  8000.1  defines 
a  functional  area  (e.g.,  personnel)  as  comprised  of  one  or  more  functional  activities  (e.g., 
recruiting),  each  of  which  consists  of  one  or  more  functional  processes  (e.g.,  interviews).  This 
information  will  be  used  in  determining  the  functional  objectives  that  the  effort  supports.  Record 
answers  on  the  worksheet,  along  with  the  name  of  the  director  and  title  of  the  sponsoring  office. 

Next,  identify  the  objectives  for  the  functional  area  that  the  project  supports,  along  with 
benchmark,  target,  and  current  estimate  for  each  objective.  In  this  example,  the  Department  is 
interested  in  decreasing  cost  and  cycle  time  for  providing  goods  and  services  to  the  warfighters. 
The  ASD(IRM)  should  be  aware  of  the  baseline  value  for  cycle  time  to  fill  a  requisition  for  a 
part. 

For  the  purposes  of  this  case  study,  assume  that  the  current  cycle  time  to  fill  a  request  for  parts  is 
96  hours,  and  it  costs  $250  to  administer  each  request.  Benchmarking  studies  of  the  best 
practices  within  industry  have  determined  that  leading  industrial  firms  can  process  requests 
within  48  hours  with  an  associated  cost  of  $100. 

Intermediate  values  of  $150  and  60  hours  have  been  established  as  approved  targets  based  upon 
unique  factors  within  the  government  procurement  regulations.  Without  changes  in  the  law,  it 
will  be  impossible  to  obtain  the  benchmarks  from  the  best  practices  of  industry. 

The  last  column  records  the  functional  area’s  revised  estimate  of  the  cycle  time  as  progress  is 
made  in  its  reduction.  At  the  start  of  the  HIRMS  project,  this  value  will  be  equal  to  the  baseline 
value.  As  progress  is  made  in  developing  and  deploying  HIRMS,  the  revised  estimate  will  show 
progress  towards  the  approved  target.  Worksheet  2,  illustrated  in  Exhibit  7-6,  has  been  filled  in 
to  reflect  the  baseline  and  benchmark  values  mentioned  above. 


Worksheet  2:  External  Environment 

Functional  Area:  Information  Resource  Management 


Project  Sponsor:  Assistant  Secretary  of  Defense  for  Information  Resource  Management 


Functional  Objective 

Baseline 

Value 

Benchmark 

Value 

Approved  Target 

Revised  Estimate 

1.  Cycle  Time 

96  hours 

48  hours 

60  hours 

96  hours 

2.  Cost  per  Request 

$250 

$100 

$150 

$250 

3.  Accuracy  Rate 

95% 

99% 

98% 

95% 

Return  on  Investment: 

Baseline:  4.5:1  Date:  Yesterday 

Current  Estimate:  5:1  Date:  Today 


Exhibit  7-6:  Completed  Worksheet  2 
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Associated  with  the  benchmark  and  target  values  for  the  functional  objectives  is  the  ROI  for  the 
functional  area.  ROI  for  the  IRM  functional  area  was  calculated  as  part  of  a  FEA  that  reflected 
the  functional  area’s  strategic  planning  and  BPR  activities.  The  baseline  value  of  4.5:1  was 
established  after  considering  the  HIRMS  and  other  projects  within  the  functional  area.  The 
baseline  value  of  the  ROI  is  recorded  on  the  worksheet  along  with  updates  that  occur  in 
conjunction  with  revised  economic  analyses  conducted  for  the  projects  within  the  functional  area. 

7.2.2  Step  2:  Define  IT  Effort’s  Internal  Baseline  with  Performance  Measures 


Worksheet  3: 

Performance  Measures 

Internal 

W 

Worksheets  1  and  2 

- ► 

STEP  2 

Define  TT  Effort's 

Requirements 

- ► 

Source  Documentation 

t  i  n 

.. 

- ► 

internal  r>a!i 

>eline 

W 

Worksheet  4: 

Targets  and 

T 

Thresholds 

Go  to  STEP  3 

Exhibit  7-7:  Step  2 


Once  objectives  are  identified,  select  a  set  of  performance  measures  that  gauge  the 
accomplishments  of  your  objectives.  This  provides  a  framework  to  determine  whether  the  effort 
is  heading  towards  delivering  its  benefit  to  the  DoD  community.  For  certain  specific  areas  of 
IRM,  a  minimum  set  of  essential  measures  exists  that  provides  a  starting  point  from  which  to 
create  a  full  set  of  performance  measures.  The  activities  involved  in  this  step  are  listed  below: 

1)  Identify  relevant  minimum  essential  performance  measures. 

2)  Identify  effort- specific  performance  measures. 

3)  Determine  targets  and  thresholds  for  identified  performance  measures. 

Activity  1:  Identify  relevant  minimum  essential  performance  measures. 

The  office  of  the  ASD(C3I)  has  defined  nine  IRM  areas  of  special  interest  as  discussed  in 
Section  6.  The  initial  activity  in  this  step  is  to  identify  which  of  the  nine  IRM  areas  the  HIRMS 
supports.  By  identifying  the  IRM  areas,  we  obtain  a  minimum  set  of  essential  measures  that 
were  presented  in  Section  3.  The  HIRMS  is  associated  with  the  AIS  Development  IRM  area,  and 
this  is  listed  in  the  worksheet  under  project  type.  The  identification  of  the  IRM  area  also 
indicates  which  staff  members  within  ASD(C3I)  will  be  interested  in  the  performance  of  the 
program. 

Benefits  and  quality  measures  should  relate  to  specific  targets,  such  as  organizational  goals, 
objectives,  missions,  and  functions,  which  are  directly  related  to  the  system  implementation  and 
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the  costs  incurred.  Initially,  the  list  may  also  include  items  that  cannot  be  quantified.  Qualitative 
benefits  should  not  be  dismissed  because  they  may  relate  to  important  outcomes  of  the  program. 
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Worksheet  3:  Internal  Requirements 

Project  Type:  AIS  Development 

Number: 

Name: 

Definition  and  Source: 

1. 

ROI 

ROI  of  project  taken  from  the  economic  analysis  for  the 

Milestone  II  review.  The  ratio  uses  the  standard 
measures  of  cost  and  benefits  (Measures  5  and  6.) 

2. 

Funding 

Funding  in  support  of  ASD(IRM)’s  strategic  plan  contained 
in  Exhibit  43. 

3. 

Standards 

Degree  of  compliance  with  the  TAFIM  contained  in  the 

HIRMS  architecture. 

4. 

Standard 

Data 

Number  and  percentage  of  data  elements  in  HIRMS ’s  data  model 
that  are  standardized  data  elements  from  the  DoD 

Data  Dictionary  System  (DDDS). 

5. 

Benefits 
($  value) 

Value  of  benefits  documented  in  the  economic  analysis 
developed  for  HIRMS ’s  Milestone  II  review. 

6. 

Cost 

Program  costs  documented  in  the  economic  analysis 
developed  for  HIRMS ’s  Milestone  II  review. 

7. 

Cycle 

Time 

Reduction  in  administrative  lead  time  as  documented  in  the 
benefits  portion  in  the  economic  analysis. 

8. 

Quality 
(5  measures) 

Accuracy,  response  time,  availability,  maintainability, 
and  restoration.  Separate  targets  and  thresholds  will  be 
established  based  upon  the  TEMP,  and  data  will  be  obtained 
through  project  and  developmental  testing. 

9. 

Customer 

Satisfaction 

Ease  of  system  use.  Preliminary  results  can  be  obtained 
through  developmental  and  project  testing  observations. 

10. 

Budget 

Measure  of  budget  execution,  and  rate  of  expenditures 
compared  to  projections  in  Exhibit  43. 

11. 

12... A 

Schedule 

Measure  of  schedule/time-frame  adherence  to  baseline 
established  at  Milestone  II  review. 

Exhibit  7-8:  Worksheet  3 
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Activity  2:  Identify  effort-specific  (custom)  performance  measures. 

The  second  activity  for  this  step  is  to  determine  additional  (custom)  performance  measures  that 
are  above  the  minimum  essential  set  and  describe  the  unique  characteristics  of  the  IRM  effort. 
The  individual  measures  that  are  relevant  to  the  HIRMS  are  entered  into  the  Worksheet  3  under 
the  “Custom”  heading,  as  illustrated  in  Exhibit  7-9. 

The  custom  benefit  performance  measures  listed  for  HIRMS  are  derived  from  the  objectives 
listed  in  Worksheet  1.  There  are  no  custom  cost  or  schedule  performance  measures.  The 
baseline  for  the  program  will  contain  more  detail  of  the  events  scheduled  and  the  appropriations 
used  for  program  funding. 


Worksheet  3:  Internal  Requirements 

Project  Type:  AIS  Development 

Number: 

Name: 

Definition  and  Source: 

1. 

ROI 

ROI  of  project  taken  from  the  economic  analysis  for  the 

Milestone  II  review.  The  ratio  uses  the  standard 
measures  of  cost  and  benefits  (Measures  5  and  6.) 

2. 

Funding 

Funding  in  support  of  ASD(IRM)’s  strategic  plan  contained 
in  Exhibit  43. 

3. 

Standards 

Degree  of  compliance  with  the  TAFIM  contained  in  the 

HIRMS  architecture. 

4 

Standard 

Data 

Number  and  percentage  of  data  elements  in  HIRMS ’s  data  model 
that  are  standardized  data  elements  from  the  DoD 

Data  Dictionary  System  (DDDS). 

5. 

Benefits 
($  value) 

Value  of  benefits  documented  in  the  economic  analysis 
developed  for  HIRMS’s  Milestone  II  review. 

6. 

Cost 

Program  costs  documented  in  the  economic  analysis 
developed  for  HIRMS’s  Milestone  II  review. 

7. 

Cycle 

Time 

Reduction  in  administrative  lead  time  as  documented  in  the 
benefits  portion  in  the  economic  analysis. 

8. 

Quality 
(5  measures) 

Accuracy,  response  time,  availability,  maintainability, 
and  restoration.  Separate  targets  and  thresholds  will  be 
established  based  upon  the  TEMP,  and  data  will  be  obtained 
through  project  and  developmental  testing. 

—  Continued  — 
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9. 

Customer 

Satisfaction 

Ease  of  system  use.  Preliminary  results  can  be 
obtained  through  developmental  and  project 
testing  observations. 

10. 

Budget 

Measure  of  budget  execution,  and  rate  of  expenditures 
compared  to  projections  in  Exhibit  43. 

11. 

Schedule 

Measure  of  schedule/time-frame  adherence  to  baseline 
established  at  Milestone  II  review. 

12. 

Labor 

Decrease  in  labor  efforts  associated  with  processing 
actions.  Target  established  in  FEA. 

13. 

Security 

Protection  from  authorized  access.  Requirements  come 
from  the  HIRMS  Security  Plan. 

14. 

Readiness 

Increased  readiness  from  improved  availability  of  supplies. 

15. 

Training 

Percentage  of  personnel  able  to  achieve  certification. 

Exhibit  7-9:  Completed  Worksheet  3 


The  appropriations  required  by  the  program  to  complete  the  development  phase  are: 

•  Construction 

•  Research  and  Development 

•  Procurement 

•  Operations  and  Maintenance. 

The  milestones  listed  below  represent  the  major  events  between  Milestone  II  (Development 
Decision)  and  Milestone  III  (Production  Decision)  for  HIRMS.  For  purposes  of  this  case  study, 
assume  that  the  HIRMS  program  is  currently  undergoing  project  test  and  evaluation. 

Milestones: 

•  Milestone  II  Decision  Meeting 

•  Award  Contract 

•  Begin  Validation  and  Acceptance 
Testing 

•  Complete  Validation  and  Acceptance 
Testing 

•  Begin  Project  Test  and  Evaluation 

•  Complete  Project  Test  and  Evaluation 

•  Milestone  IK  Decision  Meeting 
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The  performance  measures  listed  in  Worksheet  3  provide  the  input  to  Worksheet  4,  in  which 
target  and  threshold  values  are  added  to  the  performance  measures. 

7.2.3  Activity  3:  Determine  targets  and  thresholds  for  identified  performance  measures. 

During  this  step,  the  target  and  threshold  columns  of  the  Performance  Measures  Worksheet  will 
be  completed  for  all  measures  identified  in  Worksheet  3.  The  values  for  the  target  and  threshold 
are  based  upon  the  MAOPRs  contained  in  the  TEMP  and  policy  established  for  program 
deviations  in  DoD  Regulation  5000-2R. 

7.2.4  Benefits 

Exhibit  7-10  contains  candidate  benefit  measures  for  the  HIRMS,  along  with  target  and  threshold 
values.  Threshold  values  for  benefits  are  set  by  agreement  between  the  PM  and  program 
sponsor.  The  threshold  represents  the  level  of  performance  that  is  the  upper  limit  of  acceptable 
performance. 


Measure 

Target 

Threshold 

ROI 

6:1 

3:1 

Decreased  administrative  lead 

95%  of  actions  transmitted  within 

95%  of  actions  transmitted 

time 

72  hours 

within  96  hours 

Use  of  standard  data 

50%  of  data  elements 

25%  of  data  elements 

Use  of  standards 

Fully  compliant  with  TAFIM 

75%  compliant 

Decreased  labor  efforts 
associated  with  processing 
actions 

75%  decrease  in  labor  efforts 

50%  decrease  in  labor 
efforts 

Correct  response  to  requests  for 
status  of  pending  actions 

99% 

97% 

Response  time 

3  sec.  maximum 

4  sec.  maximum 

Ease  of  use 

75%  satisfaction 

65%  satisfaction 

Protection  from  unauthorized 

access 

100% 

98% 

Decreased  training  requirements 

90%  achieve  certification 

75%  achieve  certification 

Availability 

98% 

95% 

Mean  time  to  restore  function 

99%  within  24  hours 

99%  within  48  hours 

Restoration  of  databases 

100% 

99% 

Increased  readiness  from 

95%  demands  met  within  24 

85%  demands  met  within 

improved  availability  of  supplies 

hours 

24  hours 

Exhibit  7-10:  Candidate  Benefits  for  the  HIRMS 
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7.2.5  Costs 

Exhibit  7-11  lists  the  total  funding  by  appropriation  required  by  the  program  to  complete  the 
development  phase.  The  “Approved”  column  represents  the  dollars  shown  in  the  budget  and  are 
the  program’s  targets.  The  threshold  is  set  at  1 15%  of  the  target  or  approved  funding  level.  The 
numbers  in  the  exhibit  are  in  then-year  dollars. 


Appropriation 

Approved  (Target) 

Threshold 

Construction 

$  1.0M 

$1.15M 

Research  and  Development 

$  9.0M 

$10.35 

Procurement 

$  2.0M 

$2.3M 

Operations  and  Maintenance 

$  3.0M 

$3.45M 

Total 

$15.0M 

$17.25M 

Exhibit  7-11:  Target  and  Threshold  Funding  for  Development  Phase 


7.2.5.1  Schedule 

Exhibit  7-12  contains  the  target  and  threshold  date  for  the  major  events  for  HIRMS  during  its 
current  phase  of  its  life  cycle.  The  threshold  values  are  computed  by  adding  90  days  to  the  target 
value. 


Milestone 

Target 

Threshold 

Milestone  II  Decision  Meeting 

15  July  1996 

15  October  1996 

Award  Contract 

15  March  1997 

15  June  1997 

Begin  Validation  and  Acceptance 
Testing 

15  September  1997 

15  December  1997 

Complete  Validation  and 

Acceptance  Testing 

15  March  1998 

15  June  1998 

Begin  Project  Test  and  Evaluation 

15  April  1998 

15  July  1998 

Complete  Project  Test  and 

Evaluation 

15  July  1998 

15  October  1998 

Milestone  III  Decision  Meeting 

15  September  1998 

15  December  1998 

Exhibit  7-12:  Schedule  Milestones  for  the  HIRMS 

The  combination  of  Exhibits  7-10,  7-11,  and  7-12  are  equivalent  to  a  completed  Worksheet  4. 
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7.2.6  Step  3:  Validate  Feasibility  of  Performance  Measures 


Worksheet  5: 

STEP  3 

Validation  Worksheet 

Worksheets  3  and  4 

Validate  Feasibility 

(One  copy  for  each 

of 

performance  measure) 

Performance  Measures 

1 

Go  to  Step  4 

Exhibit  7-13:  Step  3 

Once  the  preliminary  list  of  performance  measures  is  identified,  you  evaluate  each  measure  to 
ensure  that  its  collection,  verification,  and  validation  is  possible  and  that  the  collection  is  cost 
effective.  The  best  performance  measures  are  output  from  the  process  measured.  During  this 
step,  when  an  unfeasible  performance  measurement  is  found,  refine  the  measure  or  return  to 
STEP  2  to  select  a  new  performance  measure  to  gauge  that  objective. 

The  baseline  must  be  evaluated  from  two  perspectives:  management  effectiveness  and  the  costs 
of  collecting  the  data.  This  section  illustrates  evaluating  the  management  effectiveness  criteria 
and  the  feasibility  of  collecting  the  data  for  the  measures  in  the  benefits  portion  of  the  baseline. 
Worksheet  5,  illustrated  in  Exhibit  7-14,  is  completed  for  each  performance  measure  during  this 
step  using  Worksheets  3  and  4  as  inputs.  Worksheet  5  evaluates  three  major  issues: 

1.  What  data  are  necessary  for  calculation  of  the  performance  measure,  when  are  the  data 
collected,  and  who  collects  the  data? 

2.  How  will  results  be  verified  and  validated  to  ensure  results  are  accurate? 

3.  What  is  the  cost  of  the  data  collection? 

The  answers  to  these  three  questions  will  determine  whether  this  measure  is  cost  effective  to 
collect.  If,  from  answering  these  questions,  it  is  found  that  the  measure  is  not  cost  effective, 
either  reconsider  the  means  of  collecting  the  measure  or  return  to  Step  2  and  identify  a  different 
performance  measure  that  can  more  effectively  gauge  this  objective. 

The  activities  of  this  step  are  repeated  for  each  identified  performance  measure.  The  activities 
involved  in  this  step  are  as  follows: 

1.  Identify  data  required  to  calculate  the  performance  measure  and  when  and  by  whom  the 
data  are  collected. 
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2.  Identify  the  verification  and  validation  strategy  for  the  data  collection. 

3.  Identify  the  collection  cost. 

4.  Evaluate  whether  the  performance  measurement  is  feasible.  If  not,  refine  or  delete  the 
performance  measure.  (Do  not  delete  minimum  essential  performance  measures.) 

Activity  1:  Identify  data  required  to  calculate  the  performance  measure  and  when 
and  by  whom  the  data  are  collected. 

The  following  questions  are  criteria  for  determining  feasibility: 

1.  Are  we  measuring  the  right  things?  (All  measurements  are  linked  to  an  objective  and  all 
objectives  are  measured.) 

2.  Do  we  have  the  right  measures?  (Management  decisions  can  be  made  based  on  the 
measurements,  and  effectiveness  and  efficiency  are  represented.) 

3.  Who  is  responsible  for  collecting  the  data?  (Identify  an  individual  who  is  responsible  for 
collecting  the  data) 

4.  What  is  the  cost  of  collection?  (Determine  whether  the  value  of  the  data  collected  is 
worth  the  cost  of  collection.) 

Activity  2:  Identify  the  verification  and  validation  strategy  for  the  data  collection. 

Verification  and  validation  of  measures  are  critical.  It  is  important  that  you  have  confidence  in 
the  measures  reported.  Identify  a  method  to  can  increase  the  quality  of  the  information. 

Activity  3:  Identify  the  collection  cost. 

Determine  the  administrative  burden  of  collecting  the  measurement  (hours  or  dollars).  For 
example,  if  the  measure  is  captured  in  a  standard  generated  report,  the  costs  will  be  relatively 
low. 

For  HIRMS,  we  will  validate  Measure  4  (response  time).  The  data  collected  will  be  based  upon 
the  response  times  observed  during  the  developmental  and  project  testing.  This  information  is 
entered  into  Worksheet  5. 

Activity  4:  Evaluate  whether  the  performance  measurement  is  feasible.  If  not,  refine  or 
delete  the  performance  measure.  (Do  not  delete  minimum  essential  performance 
measures.) 


7-18 


Based  on  all  of  the  information  on  Worksheet  5  and  the  benefits  and  costs  of  collecting  the 
performance  measure,  determine  whether  the  performance  measure  is  feasible. 

W orksheet  5 :  V alidation  W orksheet 

Measure  #:  4  Measure  Name:  Response  Time.  Time  required  to  receive  a 

response  after  entering  a  command. 

Data  Required:  Response  time  for  each  command  that  can  be  entered  into  the  system. 

How  Collected:  Recorded  during  developmental  and  project  testing. 

Verification/Validation  Strategy:  The  procedures  for  collecting  and  evaluating  the  data  are 
contained  in  the  Test  and  Evaluation  Master  Plan  along  with  the  test  scripts  and  data  sets. 

Collection  Cost:  The  cost  is  a  small  component  of  the  costs  of  conducting  the  required 
developmental  and  project  costing. 

Cost  Effective  Collection:  Yes 

Exhibit  7-14:  Completed  Worksheet  5 

The  performance  measure,  response  time,  is  judged  to  be  cost  effective  to  collect.  The  measure 
will  therefore  be  added  to  the  program’s  baseline. 

7.2.7  Step  4:  Finalize  Performance  Measurement  Baseline 


Worksheets  1  and  2 

Worksheet  6 A: 

STEP  4 

Quality  Checklist 

Worksheets  3  and  4 

- - - 

Finalize  Performance 

Worksheet  6B: 

Measurement  Baseline 

Objectives  Coverage 

Worksheet  ^ 

Exhibit  7-15:  Step  4 

You  now  have  a  list  of  performance  measures  that  are  linked  to  your  mission  and  objective  and 
are  cost  effective  to  collect.  The  next  step  is  to  evaluate  the  performance  measures  as  a  set  to 
determine  that  you  and  your  supervisors  can  use  them  to  determine  progress  and  track 
accomplishment  of  your  IRM  effort’s  mission  and  objectives.  During  this  step,  you  may  need  to 
delete  performance  measures  or  return  to  STEP  2  to  define  new  measures. 
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This  step  collects  the  results  of  the  previous  steps  in  a  single  document  that  is  submitted  to  the 
oversight  authority  for  approval.  This  step  establishes  a  performance  agreement  between  the  PM 
for  HIRMS  and  his  chain  of  command. 

Finalizing  the  program’s  baseline  involves  four  activities: 

1.  Ensure  that  the  set  of  performance  measures  is  measuring  the  right  thing.  Refine  set  of 
measures,  if  necessary. 

2.  Ensure  that  the  set  of  performance  measures  has  the  right  measures.  Refine  the  set  of 
measures,  if  necessary. 

3.  Gain  consensus  from  chain  of  command  for  the  performance  measurement  baseline. 

4.  Establish  data  collection  efforts  to  obtain  periodic  values  of  the  measures  in  the  baseline. 

Activity  1:  Ensure  that  your  set  of  performance  measures  is  measuring  the  right  thing. 
Refine  set  of  measures,  if  necessary. 

This  activity  is  accomplished  by  reviewing  the  questions  in  Worksheet  6A,  as  illustrated  in 
Exhibit  7-16,  and  making  adjustments  to  the  set  of  performance  measures  until  the  answer  to 
each  question  is  “yes”. 


Worksheet  6A:  Quality  Checklist 


YES 

NO 

1.  Are  we  measuring  the  right  thing? 

Does  the  set  of  measures  address  improvement  in  performance  of  objectives? 

X 

Does  the  set  of  measures  use  a  small  set  of  significant  performance  measures  that 
provide  a  clear  basis  for  assessing  accomplishment,  facilitate  decision-making,  and 
focus  on  accountability? 

X 

Does  the  set  of  measures  assess  the  “value-added”  contribution  made  by  the  IRM 
investment? 

X 

Does  the  set  of  measures  capture  the  requirements  of  internal  and  external  customers? 

X 

Does  the  set  of  measures  address  the  external  performance  of  the  functional  area? 

X 

Does  the  set  of  measures  address  the  benefits,  costs,  and  schedules? 

X 

2.  Do  we  have  the  right  measures? 

Are  most  measures  linked  to  a  clear  outcome  (results  rather  than  inputs  or  outputs)? 

X 

Is  the  set  of  measures  understood  at  all  levels  that  have  to  evaluate  and  use  the 
measures? 

X 

Is  the  set  of  measures  effective  in  prompting  action? 

X 

Is  the  set  of  measures  accurate,  reliable,  valid,  verifiable,  and  cost-effective? 

X 

Does  the  set  of  measures  include,  along  with  long-term  measures,  short-term  measures 
or  goals  that  show  interim  progress? 

X 

Exhibit  7-16:  Worksheet  6A 
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Activity  2:  Ensure  that  the  set  of  performance  measures  has  the  right  measures.  Refine 
the  set  of  measures,  if  necessary. 


A  final  worksheet,  as  illustrated  in  Exhibit  7-17,  maps  objectives  to  measures  and  is  completed 
during  this  activity  to  ensure  coverage  of  all  objectives. 

Worksheet  6B:  Objectives  Coverage  Worksheet 


Performance 

Measure 

Decreased 

Lead 

Time 

Decreased 

Cost 

Accuracy, 
Response  Time, 
Availability, 
Maintainability, 
Restoration 

Security 

User- 

Friendly 

Training, 

Sustainable 

with 

Current 

Personnel 

ROI 

X 

Funding 

X 

Standards 

Standard 

Data 

Benefits 

X 

Costs 

X 

Cycle  Time 

X 

Quality 

X 

Customer 

Satisfaction 

X 

Budget 

X 

Schedule 

Labor 

X 

Security 

X 

Readiness 

X 

Training 

X 

Exhibit  7-17:  Worksheet  6B 

Activity  3:  Gain  consensus  within  chain  of  command  for  the  performance 
measurement  baseline. 

The  baseline  document  is  a  formal  agreement  between  the  program  functional  proponent  and  the 
DoD  Executive  Agent.  The  PM  is  responsible  for  preparing  the  baseline  for  approval  within  the 
functional  organization  responsible  for  the  program  and  the  Services’  or  Agencies’  Executive 
Agent.  We  will  assume  that  the  PM  for  the  HIRMS  has  updated  his  baseline  prior  to  gaining  the 
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Milestone  II  approval.  This  baseline  now  forms  the  contract  against  which  the  PM  manages  the 
program.  (See  Appendix  A  for  proposed  baseline  agreement  format.) 


Activity  4.  Establish  data  collection  efforts  to  obtain  periodic  values  of  the  measures 
in  the  baseline. 

The  PM  is  now  responsible  for  gathering  sufficiently  accurate  and  timely  data  to  be  able  to  judge 
whether  his  program  is  executing  within  the  approved  parameters  for  cost,  schedule,  and 
performance.  Testing  events,  such  as  developmental  and  project  testing,  will  provide  leading 
indicators  of  how  the  system  will  ultimately  perform.  Data  must  be  collected  and  processed  to 
compute  “current  estimates”  of  the  values  for  the  measures  in  the  baseline. 

For  the  purposes  of  this  case  study,  we  will  illustrate  breach  conditions  for  the  HIRMS  in  the 
areas  of  cost,  schedule,  and  benefit.  The  current  estimates,  based  upon  data  collected,  indicate 
that  there  are  variances  between  the  program’s  targets  and  the  current  estimates. 

A  baseline  breach  occurs  when  the  program  deviates  from  the  approved  baseline.  A  breach  of 
the  baseline  occurs  when  the  cost  shown  in  the  baseline  agreement  is  estimated  to  increase  by 
more  than  15%  during  the  project  development  phase;  there  is  a  projected  schedule  slippage  of 
more  than  three  months;  or  there  are  modifications  to  approved  program  funding  that  result  in  a 
nonexecutable  baseline. 

The  HIRMS  program  is  in  the  developmental  phase  between  Milestones  II  and  HI  and  is 
currently  going  through  project  testing.  Several  technical  parameters  of  the  benefits  are  not  at 
desired  levels.  The  program  is  currently  facing  a  slip  in  schedule  due  to  software  redesign,  and 
the  costs  of  completing  the  current  phase  are  also  expected  to  rise. 

Exhibit  7-18  lists  the  total  funding  by  appropriation  required  by  the  program  to  complete  the 
development  phase.  The  “Approved”  column  represents  the  dollars  shown  in  the  budget.  The 
“Required”  column  represents  what  the  PM  currently  estimates  is  needed  for  meeting  the 
approved  baseline.  The  numbers  in  the  exhibit  are  in  then-year  dollars. 


Appropriation 

Approved 

Required 

Construction 

$  1.0M 

$  1.0M 

Research  and  Development 

$  9.0M 

$  12.0M 

Procurement 

$  2.0M 

$  2.0M 

Operations  and  Maintenance 

$  3.0M 

$  4.0M 

Total 

$15.0M 

$19.0M 

Exhibit  7-18:  Approved  and  Required  Funding  for  Development  Phase  of  the  HIRMS 

The  PM  currently  requests  $4.0  million  more  than  is  budgeted  for  the  development  phase  of  the 
life  cycle.  This  represents  a  potential  cost  growth  of  26.6%.  Since  the  allowable  tolerance  for 
cost  growth  is  15%,  the  program  has  breached  its  cost  baseline,  and  corrective  actions  are 
required  to  bring  the  program  back  under  the  budgeted  costs.  If  the  cost  growth  is  significant 
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enough,  the  program’ s  expected  ROI  must  be  recalculated  to  ensure  that  the  program  is  still  a 
good  investment  according  to  the  agency’s  investment  portfolio. 

Exhibit  7-19  contains  the  schedule  milestones  for  the  HIRMS.  The  “Completed”  column 
represents  the  dates  on  which  the  events  were  actually  completed.  The  “Modified  Schedule” 
column  represents  the  current  estimate  of  when  the  event  will  occur. 


Milestone 

Approved 

Schedule 

Completed 

Modified  Schedule 

Milestone  II 

15  July  1996 

15  July  1996 

Award  Contract 

15  March  1997 

15  March  1997 

Begin  Validation  and 
Acceptance  Testing 

15  September  1997 

15  September  1997 

Complete  Validation 
and  Acceptance 

Testing 

15  March  1998 

15  March  1998 

Begin  Project  Test  and 
Evaluation 

15  April  1998 

15  April  1998 

Complete  Project  Test 
and  Evaluation 

15  July  1998 

15  October  1998 

Milestone  HI 

15  September  1998 

15  December  1998 

Exhibit  7-19:  Schedule  Milestones 

The  program  is  currently  estimating  that  the  Milestone  HI  review  will  slip  by  over  90  days  due  to 
the  project  testing  taking  longer  than  anticipated.  This  correlates  with  the  estimated  increases  in 
program  costs  due  to  required  redesign  and  additional  testing.  The  program  has  breached  its 
schedule  baseline,  and  management  review  is  warranted. 

Exhibit  7-20  below  contains  results  from  the  HIRMS  project  test  and  evaluation.  The  number  of 
breaches  (in  bold)  indicates  that  additional  work  is  required  during  the  current  phase  of  the 
program  to  obtain  the  desired  level  of  performance  in  the  benefits  area. 

One  method  of  demonstrating  that  a  program  is  on  track  at  a  particular  point  in  time  is  to 
compare  the  current  estimates  of  the  components  (cost,  schedule,  and  benefit)  within  the  baseline 
against  the  target  and  threshold  values.  The  trend  of  a  particular  performance  measure  is 
extremely  important  for  evaluating  anticipated  results  and  the  direction  of  movement.  Current 
estimates  that  exceed  the  threshold  value  indicate  a  breach  condition  where  corrective  action  is 
warranted.  Current  estimates  between  the  target  and  threshold  indicate  a  warning  condition  that 
performance  is  not  at  the  desired  level,  and  steps  should  be  taken  to  ensure  that  the  values  do  not 
extend  below  the  threshold. 
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Measure 

Target 

Threshold 

Observed  Value 

Decreased  administrative 

95%  of  actions 

95%  of  actions 

95%  of  actions 

lead  time 

transmitted  within 

72  hours 

transmitted  within 

96  hours 

transmitted 

within  96  hours 

Decreased  labor  efforts 
associated  with 
processing  management 
actions 

75%  decrease  in 
labor  efforts 

50%  decrease  in 
labor  efforts 

N/A 

Correct  response  to 
requests  for  information 

99% 

97% 

98% 

Response  time 

3  sec.  maximum 

4  sec.  maximum 

5  sec. 

Ease  of  use 

75%  satisfaction 

65%  satisfaction 

N/A 

Protection  from 
unauthorized  access 

100% 

98% 

99% 

Decreased  training 
requirements 

90%  achieve 
certification 

75%  achieve 
certification 

N/A 

Availability 

98% 

95% 

85% 

Mean  time  to  restore 

99%  within  24 

99%  within  48 

95%  within  48 

function 

hours 

hours 

hours 

Restoration  of  databases 

100% 

99% 

99% 

Increased  readiness  from 
improved  availability  of 
information 

95%  demands  met 
within  24  hours 

85%  demands  met 
within  24  hours 

N/A 

Number  of  shared  data 
elements 

400 

300 

350 

Reduction  in  system 
development  cost  due  to 
use  of  standard  data 
elements 

$30K  reduction 

$25  K  reduction 

$26K  reduction 

ROI 

3.5 

3.0 

3.2 

Savings  due  to  reduction 

$200K  reduction 

$90K  reduction 

$250K  reduction 

in  paperwork  and 
increased  accuracy  in 
information  provided  to 
management 

over  three  years 

over  three  years 

over  three  years 

Savings  due  to 

$500K  reduction 

$45  OK  reduction 

$45  3  K  reduction 

reengineering  of  business 
practices 

over  three  years 

over  three  years 

over  three  years 

Increased  timeliness  and 
accuracy  of  responses  due 
to  BPR  initiated  changes 

Less  than  1% 
reject  rate  per  year 

Less  than  2% 
reject  rate  per  year 

1.9%  reject  rate 

Exhibit  7-20:  Candidate  Benefits  for  the  HIRMS 
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7.3  Investment  Baseline/Performance  Agreement  For  The  HIRMS 


The  information  collected  on  the  worksheets  through  the  process  described  in  sections  7.1  and 
7.2  was  used  to  complete  the  following  Investment  Baseline/Performance  Agreement  for  the 


HIRMS. 


OPERATIONAL  CAPABILITY 


Program  Name:  Hypothetical  Information  Resources  Management  System  (HIRMS) 

Mission  Goal(s):  Information  Resources  Management  (IRM)  Functional  Area 

Enhance  customer  service  through  process  improvements,  elimination  of  paperwork,  and 
improved  automated  tools  for  system  users.  Incorporation  of  improved  and  standardized 
business  practices  and  electronic  commerce  techniques.  Improve  customer  satisfaction  by 
reducing  costly,  time-consuming  paperwork;  increasing  responsiveness  to  customer  inquiries; 
and  facilitating  prompt  and  accurate  responses  to  requests  for  information. 

Program  Objectives: 

1.  Support  the  use  of  standard  department  management  policies,  processes,  and  shareable  data. 

2.  Improve  timeliness,  accuracy,  and  effectiveness  of  management  information. 

3.  Optimize,  streamline,  and  integrate  disparate  IRM  automated  systems,  subsystems,  and 
databases. 

4.  Facilitate  the  department-wide  integration  of  a  standard,  robust,  management  environment 
through  the  implementation  of  standard  processes,  and  standard  shared  data. 

5.  Provide  for  improved  data  management  and  data  integrity  by  electronic  input  of  selected  data 
to  a  logically  shared  data  repository.  Standard  data  and  data  transmissions  must  be  employed. 
The  capability  to  exchange  data  within  the  department,  other  government  agencies,  and  with 
industry  must  be  provided. 

6.  Provide  information  exchange  capabilities  among  department  components  and  related 
functional  areas. 

7.  Provide  for  use  of  department- wide  electronic  commerce/electronic  data  interchange 
(EC/EDI). 

8.  Streamline  manual  management  processes,  including  the  automation  of  manual  management 
activities  and  the  ability  to  input  data  only  once  at  the  source. 

9.  Provide  an  on-line  means  for  capturing  and  evaluating  customer  feedback  information. 

10.  Provide  the  status  of  materials  that  are  on  order  or  on  hand  in  a  near-re al-time  environment  to 
enable  department  managers  to  more  closely  monitor  the  assets  of  the  department. 

Current  Operational  Capability: 

Currently  the  DoD  IRM  Community  is  not  using  a  single  standardized  system  to  perform  the 
information  resource  management  function.  Each  component  is  using  one  or  more  “stovepipe” 
systems  to  perform  their  mission.  These  “legacy  systems”  are  not  usually  integrated  among  the 
components  and  in  only  a  few  cases  is  EC/EDI  possible.  There  is  also  a  significant  percentage  of 
activities  who  have  no  automated  IRM  support  systems. 
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Enhanced  Operational  Capability 

Requirements 

Enhanced  Operational  Capability 

Performance  Measures: 

Decreased  administrative  lead  time. 

95%  of  management  actions  completed  within  96 
hours. 

Decreased  cost  of  management  due  to 
streamlining  of  processes. 

75%  decrease  in  person-hours  required  to  process 
a  request. 

Provision  of  accurate  information  on  requests  for 
status  of  pending  actions. 

99%  accuracy. 

Protection  of  information  from  unauthorized 

access. 

Rejects  unauthorized  access  and  intrusion  100% 
of  the  time. 

User-friendly  design. 

At  least  75%  of  HIRMS  users  express 
satisfaction. 

Effective  training  program  for  personnel  who 
operate/maintain  HIRMS. 

At  least  90%  of  trained  users  achieve 
certification. 

Sustainability  under  the  current  personnel. 

No  increase  in  the  number  of  personnel  or  the 
entrance  abilities  of  user  personnel. 

Acceptable  response  time  to  user  commands. 

Response  time  after  initiating  a  command  shall  be 

3  seconds  maximum  with  a  desired  interval  of  2 
seconds. 

Maximum  availability  to  sustain  operations. 

Available  98%  of  time  for  software  and 
interfaces.  Minimum  allowable  is  95%. 

Short  mean  time  to  restore  function. 

99%  of  system  faults  shall  be  corrected  within  48 
hours. 

Provision  of  the  capability  to  rebuild  and  restore 
databases. 

100%  of  data  from  last  backup  is  recovered 

Benefits  Assessed; 

Return  on  Investment:  3:1 
Quality: 

•  Accuracy 

•  Response  Time 

•  Availability 

•  Maintainability 

•  Restoration 

Compatibility:  Compliance  with  TAFIM. 
Interoperability:  Use  of  DoD  standard  elements. 
Annual  Maintenance: 

Annual  Operations  &  Support  Costs 
Risks  &  Deficiencies: 

Failed  Performance  Goals/Measures 


7-26 


Strategic  Match: 

The  HIRMS  is  a  critical  component  in  the  IRM  Functional  Area  Strategic  Plan  to  have  a 
common  information  resources  management  throughout  DoD.  The  common  system  will  support 
the  functional  areas  goals  for  reductions  in  response  times,  errors,  and  costs. 

Conduct  Operational  Benefits  Assessment  /  Post-Deployment 

A  post-deployment  operational  benefits  assessment  will  be  conducted  beginning  180  days  after 
IOC  (May  2000.) 

PROGRAM  COST  &  SCHEDULE 


Program  Costs: 

Then  Year 
Base  Year 

Approved  Budget  Amount 

Average  Unit  Procurement  Cost 

Life  Cycle  Cost 

Sunk  Cost 

Cost  to  Complete 


$19,000,000 
$16,500,000 
$15,000,000 
N/A  (single  system) 
$60,000,000 
$5,000,000 
$55,000,000 


Program  Schedule: 

Program  Initiation 
Major  Milestone  Decision  Points 
Milestone  0 
Milestone  I 
Milestone  II 
Critical  System  Events 
Award  Contract 

Begin  Validation  and  Acceptance  Testing 
Complete  Validation  and  Acceptance  Testing 
Begin  Project  Test  and  Evaluation 
Complete  Project  Test  and  Evaluation 


March  1994 

March  1994 
April  1995 
July  1996 

15  March  1997 
15  September  1997 
15  March  1998 
15  April  1998 
15  July  1998 


Milestone  HI 

Initial  Operating  Capability 


September  1998 
October  1999 


Risks  &  Benefits: 


Risks  Assessed; 

•  Strategic  Uncertainty:  The  HIRMS  will  support  streamlining  of  existing  processes.  There 
is  minimal  risk  of  failure  in  this  area. 
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•  Technological  Uncertainty:  HIRMS  uses  technology  and  techniques  well-tried  and  proven 
in  Government  and  commercial  use.  Minimal  risk  in  this  area. 

•  IT  Infrastructure  Risks:  There  is  a  moderate  risk  in  this  area  because  of  HIRMS 
dependence  on  the  existing  data  communications  system.  If  data  traffic  from  other  sources 
significantly  exceeds  the  planning  assumptions;  HIRMS  could  fall  below  required  levels  for 
response  times. 

•  Organizational  Risks:  There  is  a  minimal  risk  in  this  area.  HIRMS  requires  significant  re¬ 
training  of  DoD  personnel  to  effectively  use  the  full  system  capabilities;  however,  analysis  of 
the  proposed  training  program  reveals  no  likely  problem  areas. 

Strategic  Impacts  : 

•  Management  Information  Assessment:  Fielding  of  the  HIRMS  directly  supports  the 
following  mission  goals  of  the  OASD  (IRM): 

1.  Enhance  customer  service  through  process  improvements,  elimination  of  paperwork,  and 
improved  automated  tools  for  system  users. 

2.  Incorporation  of  improved  and  standardized  business  practices  and  electronic  commerce 
techniques. 

3.  Improve  customer  satisfaction  by  reducing  costly,  time-consuming  paperwork;  increasing 
responsiveness  to  customer  inquiries;  and  facilitating  prompt  and  accurate  responses  to 
requests  for  information. 

•  Competitive  Response:  Failure  to  field  the  HIRMS  (or  the  same  capabilities)  will  seriously 
degrade  DoD’s  ability  to  effectively  manage  IT  resources.  Inefficient  management 
procedures,  lack  of  accountability,  and  ineffective  user  support  will  worsen  without  the 
automated  management  capabilities  HIRMS  is  designed  to  provide.  Inordinate  amounts  of 
the  IT  budget  will  have  to  be  committed  to  maintaining  the  current  system  instead  of 
capitalizing  on  mission-critical  enhancements. 

•  Strategic  Information  Systems  Architecture:  HIRMS  is  fully  compliant  with  TAFIM,  the 
DoD  data  standards,  and  the  DoD  IM/TT  Strategic  Plan. 

Variance  From  Program  Baseline  Goals 

Variance  in  total  program  cost  -  HIRMS  is  currently  projected  to  be  $4.0  million  (26.6%) 
above  baseline  costs  at  the  end  of  this  FY.  The  cost  variance  is  caused  by  unanticipated 
problems  in  software  development. 

Variance  in  total  program  schedule  -  HIRMS  is  currently  3  months  behind  the  baseline 
schedule,  again  because  of  software  development  problems. 

Variance  in  operational  capability  performance  indicators  -  HIRMS  is  currently  failing  to 
meet  performance  baselines  in  three  areas: 

1.  Response  time  (5  seconds  vs.  4  seconds) 

2.  Availability  (85%  vs.  95%) 

3.  Mean  time  to  restore  function  (95%  within  48  hours  vs.  99%  within  48  hours) 
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Corrective  Actions: 


1.  Additional  software  development  resources  were  committed  to  resolve  the  immediate 
problem  causing  the  cost  and  schedule  delays.  Two  additional  software  engineers  were  added  to 
the  project  team  and  a  more  rigorous  design  review  process  was  instituted  to  preclude  further 
slippage. 

2.  The  response  time  problem  has  proved  resistant  to  all  alternatives  explored.  It  is  the 
consensus  of  the  project  technical  team  that  the  current  value  represents  the  best  performance 
available  with  current  technology.  Recommend  relaxing  the  performance  requirement. 

3.  The  availability  and  mean  time  to  restore  function  failures  have  required  minor  redesigns  of 
two  modules.  Preliminary  testing  indicates  these  problems  can  be  eliminated  without  cost  or 
schedule  impact.  Expect  confirmation  of  test  results  in  next  30  days. 

Proposed  Revisions  to  Baseline  Goals: 

The  required  response  time  of  4  seconds  has  proven  to  be  technically  infeasible  given  current 
technology  and  the  infrastructure  constraints  of  the  system.  5  seconds  has  consistently  been  the 
best  the  system  can  deliver,  and  this  has  not  responded  to  any  alternative  solutions.  User 
representatives  on  the  Integrated  Project  Team  have  indicated  that  5  seconds  is  operationally 
adequate  if  it  can  be  consistently  delivered.  Recommend  relaxing  this  requirement  to  5  seconds. 
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ACQUISITION/CONTRACT  BASELINE 


Earned  Value  Management  Framework 


Budgeted  Cost  of  Work  Performed: 
Budgeted  Cost  of  Work  Scheduled: 
Actual  Cost  of  Work  Performed: 
Actual  Cost  of  Work  Scheduled: 


$7,500,000 

$15,000,000 

$10,000,000 

$19,000,000 


Cost  Variance: 
Schedule  Variance: 


$2,500,000 
3  months 


Budget  at  Completion:  $60,000,000 

Estimate  at  Completion:  $65,000,000 


Acquisition  Baseline  Cost  &  Schedule  Goals  {show  the  dollar  amount  of  the  project  that  will 
be  completed  each  year.  Identify  and  discuss  how  many  months  it  will  take  to  complete  the 
acquisition,  important  components,  and  important  milestones  within  that  time} 

FY  19PY  Accomplishments 
FY  19CY  Planned  Program 
FY  19BY1  Planned  Program 
FY  19BY2  Planned  Program 
Cost  Performance  Index  (CPI) 

Schedule  Performance  Index  (SPI) 
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Acquisition  Baseline  Performance  Goals 


Measure 

Target 

Threshold 

Observed  Value 

Decreased  administrative 

95%  of  actions 

95%  of  actions 

95%  of  actions 

lead  time 

transmitted  within 
72  hours 

transmitted  within 

96  hours 

transmitted 
within  96  hours 

Decreased  labor  efforts 
associated  with 
processing  management 
actions 

75%  decrease  in 
labor  efforts 

50%  decrease  in 
labor  efforts 

N/A 

Correct  response  to 
requests  for  information 

99% 

97% 

98% 

Response  time 

3  sec.  maximum 

4  sec.  maximum 

5  sec. 

Ease  of  use 

75%  satisfaction 

65%  satisfaction 

N/A 

Protection  from 
unauthorized  access 

100% 

98% 

99% 

Decreased  training 
requirements 

90%  achieve 
certification 

75%  achieve 
certification 

N/A 

Availability 

98% 

95% 

85% 

Mean  time  to  restore 

99%  within  24 

99%  within  48 

95%  within  48 

function 

hours 

hours 

hours 

Restoration  of  databases 

100% 

99% 

99% 

Increased  readiness  from 
improved  availability  of 
information 

95%  demands  met 
within  24  hours 

85%  demands  met 
within  24  hours 

N/A 

Number  of  shared  data 
elements 

400 

300 

350 

Reduction  in  system 
development  cost  due  to 
use  of  standard  data 
elements 

$30K  reduction 

$25  K  reduction 

$26K  reduction 

ROI 

3.5 

3.0 

3.2 

Savings  due  to  reduction 

$200K  reduction 

$90K  reduction 

$250K  reduction 

in  paperwork  and 
increased  accuracy  in 
information  provided  to 
management 

over  three  years 

over  three  years 

over  three  years 

Savings  due  to 

$500K  reduction 

$450K  reduction 

$453K  reduction 

reengineering  of  business 
practices 

over  three  years 

over  three  years 

over  three  years 

Increased  timeliness  and 
accuracy  of  responses  due 
to  BPR  initiated  changes 

Less  than  1% 
reject  rate  per  year 

Less  than  2% 
reject  rate  per  year 

1.9%  reject  rate 
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Variance  from  Acquisition  Baseline  Goals 

1.  Variance  in  Acquisition  Cost:  HIRMS  is  currently  projected  to  be  $4.0  million  (26.6%) 
above  baseline  costs  at  the  end  of  this  FY.  The  cost  variance  is  caused  by  unanticipated 
problems  in  software  development. 

2.  Variance  in  Acquisition  Schedule:  HIRMS  is  currently  3  months  behind  the  baseline 
schedule,  again  because  of  software  development  problems. 

3.  Variance  in  Acquisition  Performance  goals:  HIRMS  is  currently  failing  to  meet 
performance  baselines  in  three  areas: 

•  Response  time  (5  seconds  vs.  4  seconds) 

•  Availability  (85%  vs.  95%) 

•  Mean  time  to  restore  function  (95%  within  48  hours  vs.  99%  within  48  hours) 

Proposed  Revisions  To  Acquisition  Baseline  Goals 

The  required  response  time  of  4  seconds  has  proven  to  be  technically  infeasible  given  current 
technology  and  the  infrastructure  constraints  of  the  system.  5  seconds  has  consistently  been  the 
best  the  system  can  deliver,  and  this  has  not  responded  to  any  alternative  solutions.  User 
representatives  on  the  Integrated  Project  Team  have  indicated  that  5  seconds  is  operationally 
adequate  if  it  can  be  consistently  delivered.  Recommend  relaxing  this  requirement  to  5  seconds. 


Effectiveness  and  Suitability  Performance  Indicators: 

Technical  Evaluation  Criteria 

HIRMS  conforms  with  the  Technical  Architecture  Framework  for  Information  Management 
(TAFIM),  consistent  with  the  acquisition  of  commercial  software.  HIRMS  will  use  open  systems 
architecture,  the  Defense  Information  Infrastructure  (DII),  relational  database  technology, 
standardized  data,  and  Electronic  Commerce/Electronic  Data  Interchange  (EC/EDI).  HIRMS 
will  interface  with  other  functional  areas,  including  logistics  and  finance.  The  system  will  meet 
C2  level  security  requirements. 

Operational  Evaluation  Criteria 

I.  The  Critical  Operational  Issues  (COIs)  and  measurement  criteria  are  specified  in  the  Test 
and  Evaluation  Master  Plan  (TEMP).  COIs  specified  include: 

A.  Performance.  Does  the  HIRMS  enable  users  to  complete  required  actions  at  and 
between  sites  in  accordance  with  applicable  regulations? 

B.  Usability.  Does  HIRMS  support  ease  of  use,  effectiveness  and  efficiency,  and 
user  satisfaction? 


7-32 


C.  Security.  Does  HIRMS  protect  its  information,  provide  adequate  system 
protection,  and  possess  the  ability  to  survive  unwanted  intrusion? 

D.  Training.  Does  the  contractor  provided  training  program  sufficiently  support 
HIRMS  operations? 

E.  Reliability,  Availability,  Maintainability  (RAM).  Is  HIRMS  reliable, 
maintainable,  and  available  for  operation? 

II.  Minimum  Acceptable  Operational  Performance  Requirements  (MAOPRs)  are  provided  in 
the  TEMP  dated  8  September  1995.  MAOPRs  represent  the  minimum  acceptable 
operational  effectiveness  and  suitability  characteristics,  along  with  performance 
thresholds  against  which  each  characteristic  will  be  evaluated  to  ensure  that  stated 
deficiencies/needs  will  be  corrected/satisfied. 
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8.  IT  PERFORMANCE  MEASUREMENT  METHODOLOGIES 


Successful  performance  measurements  produce  measures  that  are  significant,  linked  to  outcomes, 
correspond  to  a  baseline,  and  are  based  on  credible  information.  This  section  looks  at 
Information  Technology  (IT)  performance  from  a  broad  perspective,  identifying  a  number  of 
measures  and  measurement  methodologies  that  can  be  used  to  develop  the  measures  in  the 
baseline.  The  approaches  and  the  measures  detailed  in  this  section  are  in  no  way  a  complete  set 
of  measures  and  are  included  to  help  the  user  start  generating  performance  measures. 

The  list  of  methods  that  follows  below  is  designed  to  assist  users  of  this  guide  to  select  the  most 
appropriate  measurement  approach  for  their  needs.  Each  of  the  sections  listed  contains  a  brief 
description  of  the  approach,  an  example  of  the  generated  performance  measures,  a  discussion  of 
the  strengths  and  weaknesses  of  the  approach,  how  the  approach  might  be  used,  and  where  to 
find  more  information  about  the  approach.  It  may  be  necessary  to  use  several  of  the  approaches 
for  a  single  IT  performance  measurement  baseline. 

8.1  IT  Effectiveness  Framework 

The  IT  Effectiveness  Framework  assesses  IT  effectiveness  at  three  levels:  information  and 
support  provided;  impact  on  user  processes  and  performance;  and,  organizational  performance. 

8.1.1  Uses: 

The  framework  is  useful  for  assessing  the  effectiveness  of  IT  in  an  organization  or  unit.  It  can 
also  be  used  for  assessing  the  effectiveness  of  an  individual  system. 

8.1.2  Measurement  Approach: 

The  IT  Effectiveness  Framework  suggests  that  effectiveness  should  be  measured  at  three  levels: 

1.  Information  and  Support  Provided.  This  addresses  the  lowest  level  of  impact,  since  it 
assesses  how  effectively  IT  meets  the  information  needs  of  the  users  of  systems. 

2.  Impact  on  Processes  and  Performance.  This  level  assesses  how  well  IT  contributes  to 
improving  organizational  processes  and  their  performance. 

3.  Organizational  Performance.  At  the  highest  level,  IT  should  have  an  impact  on  some 
aspects  of  the  organization's  overall  performance. 

To  implement  the  framework,  managers  must  first  establish  the  ways  IT  contributes  to  the 
accomplishment  of  the  organization  objectives  in  the  business  unit  (e.g.,  improving  sales, 
improving  customer  satisfaction).  Since  achieving  these  goals  will  be  the  result  of  lower  level 
impacts,  objectives  for  each  of  these  must  be  established  as  well  (e.g.,  improved  decision¬ 
making,  better  information  quality). 

Once  the  expected  impacts  of  IT  have  been  identified,  performance  measures  can  be  developed 
to  determine  how  effectively  IT  is  doing  its  job.  For  example,  measures  of  improved  information 
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quality  might  include  measures  of.  data  accuracy,  its  scope,  and  the  availability  of  different  levels 
of  data  aggregation. 

Performance  evaluations  provided  by  individuals  are  good  ways  to  assess  IT  effectiveness, 
provided  a  variety  of  points  of  view  are  incorporated  and  integrated  into  the  overall  evaluation. 
Such  evaluations  should  include  contributions  from  users,  IT,  management  and  internal  audit  as  a 
minimum. 

8.1.3  Strengths: 

The  effectiveness  of  IT  has  typically  been  difficult  to  evaluate.  The  dynamic  nature  of  systems 
can  mitigate  against  the  development  of  useful  measures  of  effectiveness.  This  framework 
attempts  to  avoid  these  common  problems  by  broadening  the  range  of  performance  involved. 

8.1.4  Weaknesses: 

The  development  and  management  of  effectiveness  objectives  can  be  difficult  and  time- 
consuming. 
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8.1.5  Examples: 

Example  1.  Sample  Effectiveness-Oriented  Objectives  And  Performance  Measures 


Levels 

Objectives 

Sample  Performance  Measures 

Information  and  Support 
Provided 

Improve  time  of 
presentation 

•  Data  currency 

•  Delivery  schedule 

•  Response  time 

Improve  information 
quality 

Data  accuracy,  scope,  aggregation 

Improve  information 
quantity 

•  Access  to  new  data 

•  System  interface,  flexibility, 
simplicity,  ease  of  use 

Improve  presentation  form 

Format:  graphical,  color,  etc. 

Improve  user  support 

•  Amount  of  user  training 

•  Quality  of  user  guides 

•  Quality  of  MIS  -  user  relationship 

Impact  on  Processes  and 
Performance 

Extent  of  common 
information 

Change  in  attitudes  toward  MIS 

Improved  decision  making 
process 

•  Explicitness  of  goals/objectives 

•  Consideration  of  alternatives 

•  Comprehensiveness  of  analysis 

•  Quantification  of  action 
consequences 

•  Length  of  time  to  make  decisions 

Improved  user 
organizational  performance 
via: 

Automate  manual  data 
handling/correction 

Reduced  information 
processing  costs 

Cost  displacement(people, 
equipment) 

Improve  asset  utilization 

•  Reduced  inventory 
levels/turnaround 

•  Reduced  number  of  backorders 
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Example  2.  Sample  Innovation  and  Improvement  Measures 


Innovation 

•  New  Product  Developments 

•  Product  Performance  Characteristics 

•  Time  to  Develop  Next  Generation  of  Products 

•  Product  Launch  Times:  Schedule  Adherence 

Improvement 

Rate  of  Improvement  of  Performance 

Example  3:  Sample  Financial  Measures 


Survival 

•  Operating  Profit 

•  Net  Income 

•  Cash  Flow 

Growth 

•  Market  Share 

•  Sales  Growth 

•  Return  on  Capital  Employed 

Future  Outlook 

Forecast  and  Discounted  Future  Cash  Flows 
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8.2  IT  Efficiency  Framework 


The  IT  Efficiency  Framework  assesses  IT  efficiency  at  four  levels:  (1)  the  individual  information 
system;  (2)  the  resources  required;  (3)  the  production  capability  of  IT;  (4)  and  the  level  of 
investment  in  IT  resources. 

8.2.1  Uses: 

This  framework  can  be  used  to  assess  the  efficiency  with  which  IT  development  and  operations 
use  resources.  It  can  be  used  to  assess  the  efficiency  of  an  individual  system  or  of  IT  itself. 

8.2.2  Measurement  Approach: 

The  IT  Efficiency  Framework  suggests  that  IT  efficiency  can  be  measured  at  four  levels: 

1.  The  Individual  Information  System.  This  level  addresses  how  closely  the  individual 
system  complies  with  development  and  operations  standards  (e.g.,  the  adequacy  and 
completeness  of  controls). 

2.  Resource  Consumption.  This  level  looks  at  how  closely  the  actual  system  or  systems 
meet  resource  plans  (e.g.,  schedules  and  budgets). 

3.  Production  Capability.  Here  the  number  of  resources  available  is  assessed  (e.g., 
available  person-hours). 

4.  Resource  Investment.  At  the  highest  level,  the  organization's  investment  in  resources 
is  evaluated  (e.g.,  the  capital  investment  in  hardware). 

To  implement  the  framework,  managers  must  first  identify  objectives  for  the  individual  levels  of 
efficiency.  For  example,  adherence  to  budget  might  be  one  objective  for  level  2.  When  this  has 
been  accomplished,  performance  measures  can  be  designed  and  monitored,  e.g.,  variance  from 
budget  or  percentage  of  projects  that  varied  from  budget. 

8.2.3  Strengths: 

Finking  efficiency  objectives  and  performance  measures  ensures  that  IT  monitors  efficiency 
appropriately. 

8.2.4  Weaknesses. 

This  is  only  one  type  of  framework  that  can  be  used  to  assess  IT  efficiency.  Many  alternative 
approaches  can  be  used  effectively  also.  The  collection  of  data  about  efficiency  and  the 
development  of  appropriate  objectives  and  measures  can  be  a  time-consuming  process. 
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8.2.5  Sample  IT  Efficiency  Measures 


MIS  Development  Process 


Levels 


Information 

Systems 


Resource 

Consumption 


Production 

Capability 


Objectives 


Technical 

quality 


Controls  quality 


Documentation 

quality 


Development 

budget 


Scheduled 

completion 


User 

Participation 


Available 
person-  hours 


Sample  Performance 
Measures 


Compliance  to  systems 
development  standards  for 
program  and  database 
design 


Compliance  to 
applications  control 
standards 


Compliance  to 
documentation  standards 


Budget  variance 


Schedule  compliance 


Amount  and  type  of 
involvement 


•  Chargeable  person- 
hours 

•  Productivity  rate 

•  Percent  overtime 


Resource 

Investment 


MIS  personnel 
training 


Training  expenditures 


MIS  Operations  Process 

Objectives 

Sample  Performance 
Measures 

Technical 

quality 

Compliance  to  design 
specification 

Controls  quality 

Compliance  test  for 
adequacy  and 
completeness 

Documentation 

quality 

Compliance  to  standards 

Operations 

budget 

Budget  variance 

Scheduled  run 
times 

•  Actual  run  times 

•  Percent  reruns 

Estimated 
computer 
resource  units 
required 

Actual  resource  units 
utilized 

Available 

computer 

capacity 

(throughput) 

•  %  uptime 

•  Response  time 

•  Backlog 

•  %  utilization 

•  Actual  throughput 

Job  description 

•  Job  satisfaction 

•  Job  performance 

Capital 

investment 

Capital  expenditures 
(hardware) 
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8.3  Performance  Measures  for  IT 

The  Performance  Measures  approach  outlines  ten  dimensions  of  IT  performance.  It  emphasizes 
that  appropriate  performance  measures  can  change  over  time. 

8.3.1  Uses: 

The  categories  used  in  the  Performance  Measures  approach  will  help  IT  managers  to  select 
measures  of  interest  to  Senior  Leaderships. 

8.3.2  Measurement  Approach: 

The  ten  most  important  aspects  of  IT  performance: 

1.  IT  impact  on  strategic  direction 

2.  Integration  of  IT  planning  with  corporate  planning 

3.  Quality  of  information  outputs 

4.  IT  contribution  to  organizational  financial  performance 

5.  IT  project  efficiency 

6.  User/management  attitudes  about  IT 

7.  IT  staff  competence 

8.  Integration  with  related  technologies  across  other  organizational  units 

9.  Adequacy  of  system  development  practices 

10.  Ability  of  IT  to  identify  and  assimilate  new  technologies. 

A  variety  of  measures  can  be  selected  to  evaluate  IT  performance  in  each  of  the  above  areas. 

An  effective  program  of  IT  performance  measurement  encompasses  all  of  these  areas  of  senior 
management  interest.  However,  as  the  IT  organization  develops,  the  emphasis  on  measurement 
should  change  from  a  more  structured  focus  on  project  efficiency  and  user  satisfaction  to  more 
unstructured  measures  such  as  impact  on  strategic  direction.  IT  managers  therefore  need  to 
balance  the  various  dimensions  of  performance  measurement  according  to  the  organization's 
needs  at  a  particular  time. 

8.3.3  Strengths: 

The  dimensions  of  IT  performance  measurement  in  this  approach  are  clearly  linked  with  what 
senior  management  wants  to  know  about  IT.  This  approach  incorporates  analysis  of  both  the 
tangible  and  intangible  impacts  of  IT 

8.3.4  Weaknesses: 

This  framework  does  not  incorporate  many  dimensions  of  IT  performance  that  could  be  of 
interest,  for  example,  quality  of  user  training.  Collection,  interpretation,  and  presentation  of  such 
a  widely-varying  amount  of  performance  data  can  be  a  time-consuming  task. 
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8.3.5  Sample  Performance  Measures  for  IT 


Performance  Measure 

Measurement  Description 

IT  impact  on  strategic  direction 

•  Productivity  increases  attributable  to  IT  function 

•  Cost  reductions  attributable  to  IT  function 

•  Organization  would  be  out  of  business  without  IT 

Integration  of  IT  planning  with 
enterprise  planning 

•  IT  documented  plan  is  designed  to  support  the  enterprise 
strategic  plan 

•  Forecasts  of  IT  capabilities  exist 

•  Enterprise  and  IT  plans  jointly  developed 

Quality  of  Information  Outputs 

•  End-user  surveys  (in-house) 

•  Customer/client  surveys  (outside  organization) 

•  Log  of  errors  encountered  by  users  maintained 

IT  contribution  to  organizational 
financial  performance 

•  ROI 

•  ROA 

•  Cost  allocation  (method  of  accounting  for  systems 
operations  and  development) 

•  Value  added  by  IT  (ROM) 

•  Comparison  of  IT  budgets  as  a  percentage  of  revenue  . 

•  Budget  performance  (ability  to  meet  IT  budget) 

•  Cost  of  maintaining  systems 

IT  operational  efficiency 

•  Log  of  system  availability 

•  Users'  perceptions  surveys 

•  User  turnaround  time  (batch) 

•  Log  of  computer  and  communication  up/down  time 

•  System  response  time  (on  line) 

User/Management  attitudes 

•  Management  and  user  perceptions  of  IT  performance 

•  User  surveys  of  user  participation  in  systems  development 

•  User  surveys  of  IT  responsiveness  to  user  needs 

•  Time  for  IT  function  to  respond  to  user  complaints 

•  Complaint  logs 

IT  staff  competence 

•  Number  of  managerial  and  technical  education  programs 
for  IT  staff 

•  Career  ladder(s)  for  IT  staff  exist 

•  Lormal  performance  appraisal  system  used 

•  Level  of  education  of  IT  staff 

Integration  with  related 
technologies  across  other 
organizational  units 

User/IT  development  of  user/IT  budget 
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8.4  Productivity  Measures  for  IT 

These  Productivity  Measures  for  IT  include  assessments  of  both  the  efficiency  and  the 
effectiveness  of  IT.  This  measurement  approach  suggests  ways  of  selecting  appropriate  measures 
of  each  and  a  means  of  integrating  them  to  present  to  senior  management. 

8.4.1  Uses: 

This  approach  helps  managers  select  a  variety  of  meaningful  performance  measures  for  their  IT 
organization. 

8.4.2  Measurement  Approach: 

There  are  a  large  number  of  performance  measures  available  to  IT  managers.  No  single  measure 
will  provide  senior  management  with  what  they  need  to  know  about  IT  productivity.  Instead, 
multiple  measures  are  needed  in  the  following  categories: 

•  Personnel  Performance 

•  Managerial  Performance 

•  Development  Performance 

•  Goal  Setting 

•  Financial  Performance 

Within  each  category,  several  sub-measures  should  be  monitored,  including  quantitative  and 
qualitative  measures.  By  using  similar  scales  to  assess  each  sub-measure  (e.g.,  0  -  100). 
performance  can  be  aggregated  and  easily  compared. 

The  keys  to  effective  performance  measurement  include: 

1.  Collecting  data  in  a  consistent  manner  over  time  to  ensure  that  it  is  accurate  and  reliable. 

2.  Selecting  performance  measures  that  are  of  critical  importance  to  key  users.  Involving 
users  in  selecting  measures  is  therefore  paramount. 

Once  collected,  information  should  be  presented  to  senior  management  in  terms  that  can  be 
readily  understood.  Not  all  sub-measures  should  be  presented,  but  only  the  most  important  ones 
(i.e.,  those  representing  performance  in  critical  areas).  Graphics  are  a  powerful  way  to  integrate 
these  measures  clearly. 

8.4.3  Strengths: 

This  approach  integrates  both  the  effectiveness  and  efficiency  dimensions  of  IT  performance. 

The  emphasis  on  multiple  measures  of  IT  performance  gives  a  more  complete  assessment  of  the 
value  of  the  IT  group  than  single  measurement  approaches. 


8.4.4  Weaknesses: 

There  are  a  huge  number  of  productivity  measures  available  from  which  IT  managers  can  choose, 
within  each  category.  This  approach  does  not  assist  managers  in  which  ones  to  use. 
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8.4.5  Sample  IT  Productivity  Measures  And  Sub-Measures 


Performance  Measure 

Measurement  Description 

Personnel  Performance 

•  Technical  capabilities 

•  Business  knowledge 

•  Training 

•  Replacement  projections 

•  Job  satisfaction 

Managerial  Performance 

•  Attitude  of  senior  management 

•  Attitudes  of  users 

•  Performance  audits 

•  Perceptions  of  IT  problems 

•  Perceptions  of  IT  capabilities 

Developmental  Performance: 

•  Time  and  cost 

Quantitative 

•  Size  of  system  request  backlog 

•  System  maintenance  costs 

•  System  cost  standards 

•  SLOC/ELOC 

•  Charge  out  performance 

Developmental  Performance: 

•  Application  portfolio 

Qualitative 

•  Formal  methodology  quality 
—  Structured  design 

-  Project  control 

-  Productivity  aides 

—  Documentation  quality 

•  Team  size 

•  User  interaction 

Goal  Setting 

•  Senior  management  role  in  IT  planning 

•  IT  representation  in  planning 

•  Quality  of  planning 

•  Forecasts  of  future  technology  and  future  IT  capabilities 

Operational  Performance: 
Qualitative 

•  Back  up  performance 

•  Security  and  privacy 

•  User  interaction 

•  Complete  and  accurate  data 

•  Relevant,  timely  and  understandable  output 

•  User  friendly  operations 

Adequacy  of  system 

•  Percentage  of  projects  completed  on  time  and/or  within  budget 

development  practices 

•  Standard  methodology  for  system  analysis  and  design  exists 

•  Evaluation  of  user  and  IT  function  documentation  is  performed 

•  Estimates  of  number  of  person-years  in  backlog  of  system  development  requests 

IT  personnel 

•  Formal  reward  system  for  innovative  thinking  and  development  using  IT 

•  Number  of  technical  breakthroughs 

Operational  Performance: 

•  System  availability  and  utilization 

Quantitative 

•  Job  rerun  percentages 

•  Maintenance  performance  ratios 

Financial  Performance 

•  Budget  performance 

•  Cost  recovery 

•  Distribution  of  costs 

•  Market-based  industry  standard  costs 

•  Expense  categorization 
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8.5  Enhanced  Cost-Benefit  Analysis 

Enhanced  Cost-Benefit  Analysis  (CBA)  is  a  method  of  cost-benefit  and  cost-effectiveness 
analysis  that  provides  a  basis  for  a  systematic  and  rational  planning  process  for  IT  investments. 

It  is  broader  than  the  traditional  financial  techniques  than  have  previously  been  used  to  assess  IT 
investments  because  it  includes  several  techniques  of  measuring  intangibles. 

8.5.1  Uses: 

Enhanced  CBA  can  help  senior  management  make  rational  IT  investment  decisions. 

8.5.2  Measurement  Approach: 

CBA  consists  of  five  stages: 

1.  Enumerate  of  all  changes  that  will  result  from  an  IT  investment,  both  tangible  and 
intangible.  Changes  in  resource  use,  information  output,  and  performance  in  the 
sponsoring  business  unit,  other  units  in  the  business,  and  among  suppliers/distributors 
should  be  documented. 

2.  Measure  the  changes  identified  in  Stage  1.  Problems  can  occur  with  predicting  what  will 
happen  without  the  IT  investment  and  in  estimating  the  exact  costs  of  a  jointly  used 
resource  (e.g.,  a  piece  of  hardware  used  by  several  systems).  The  most  important  aspect 
of  this  stage  is  to  count  the  incremental  or  marginal  changes  to  be  brought  about  by  the 
investment. 

3.  Attempt  an  explicit  valuation  of  these  changes.  Where  changes  can  not  be  quantified, 
e.g.,  improved  quality  of  decision  making,  the  relevant  stakeholders  should  be  asked  for  a 
valuation. 

4.  Adjust  explicit  values  according  to  the  timing  and  uncertainty  of  their  occurrence. 

5.  Consider  the  valued  and  unvalued  impacts  together  to  arrive  at  a  final  assessment  of  an  IT 
alternative. 

8.5.3  Strengths. 

Enhanced  CBA  incorporates  many  important  principles  of  logical  IT  investment  decision¬ 
making.  It  identifies  and  measures  the  costs  and  benefits  for  each  stakeholder  involved  in  the  IT 
investment. 

8.5.4  Weaknesses: 

Cost-benefit  analysis  has  been  criticized  as:  focusing  too  much  on  efficiency  rather  than 
effectiveness;  having  a  bias  towards  investments  with  short-term  payoffs;  and,  ignoring  the 
exploratory  nature  of  strategic  IT  and  important  non-economic  issues. 
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8.5.5  Sample  Cost-Benefit  Analysis  Measurements 
Stage  1.  Enumeration 

Look  for:  Changes  in  hardware,  software,  and  personnel 

Changes  in  information  output,  e.g.,  quality,  speed,  timing 

Changes  in  enterprise  performance,  e.g.,  cost,  strategic  impact,  ability  to  exploit 

technical  development 

Look  in:  The  sponsoring  unit 

Other  organizational  departments  or  divisions 
Suppliers  and  distributors 

Stage  2.  Measurement 

If  change  is  measurable,  count  the  incremental  changes,  e.g.,  number  of  increased  transactions.  If 
not  measurable,  describe  the  change  clearly  and  completely,  e.g.,  describe  the  quality  of  the 
information  output. 

Stage  3.  Valuation 

Change  valuation  can  be: 

•  Explicit  -  e.g.,  the  coat  of  a  transaction  will  decrease  by  $5.00 

•  Quantitative  -  e.g.,  response  time  will  be  15  minutes. 

•  Imputed  -  e.g.,  users  state  this  change  will  be  worth  $500,000  to  their  operations 

Stage  4.  Adjustments 

Adjust  Explicit  Values  for: 

•  Timing  of  the  occurrence  of  the  change 

•  Uncertainty  of  the  change 

•  Assumptions,  i.e.,  many  measurements  and  values  are  based  on  certain  assumptions. 

•  Sensitivity  of  the  change  to  things  like  discount  rates  or  risk  premiums? 

Stage  5.  Combine  Impacts 

•  Consider  the  explicitly  valued  and  other  impacts  together. 

•  Make  a  decision,  or  reappraise  the  scope  of  the  project  e.g.,  to  include  another 
stakeholder's  perspective  or  redesign  the  IT  investment 

•  Avoid  "management  by  numbers".  Rational  decisions  do  not  always  need  to  be 
numerically  based. 
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8.6  Information  Economics 

Information  Economics  (IE)  is  a  two-step  process.  The  first  part  is  a  bottom-up  approach  that 
assesses  both  the  tangible  and  intangible  return  on  specific  IT  investments.  The  second  part  is  a 
top-down  approach  that  plans  the  environment  of  the  overall  information  systems  function  within 
the  context  of  the  present  and  future  business  organization.  (Only  the  first  part  is  addressed 
below.) 

8.6.1  Uses: 

IE  can  help  managers  to  justify  and  evaluate  information  systems  and  their  strategic  and 
economic  impacts.  IE  can  also  be  used  to  identify  the  most  effective  mix  of  projects  for  a  given 
IT  budget. 

8.6.2  Approach: 

IE  expands  the  concept  of  the  benefits  of  IT  beyond  the  rational  view  of  economic  benefits.  In 
addition  to  an  enhanced  view  of  return  on  investment  (ROI),  internal  rate  of  return  (IRR),  and  net 
present  value  (NPV),  IE  also  incorporates  the  following  into  its  evaluations: 

•  Strategic  Match 

•  Competitive  Advantage 

•  Management  Information  Assessment 

•  Competitive  Response 

•  Strategic  IT  Architecture 

As  well  as  looking  at  the  benefits  of  IT  investments,  IE  also  assesses  their  costs.  The  approach 
expands  the  concept  of  simple  costs  to  incorporate  the  following  into  its  evaluations: 

•  Strategic  Uncertainty 

•  Organizational  Risk 

•  Infrastructure  Risk 

•  Definitional  Uncertainty 

•  Technology  Uncertainty 

To  assess  IT  investment  projects,  managers  first  systematically  evaluate  each  project  in  terms  of 
its  identifiable  tangible  and  intangible  benefits,  and  its  risks  and  uncertainty. 

Second,  each  IT  investment  project  must  be  measured  against  the  ideal  of  maximum  tangible  and 
intangible  benefits  and  minimum  risks  and  uncertainties.  Generally,  a  worksheet  detailing  the 
criteria  for  each  class  of  benefit  or  risk  is  completed  for  each  project. 

Through  this  process,  a  project  score  is  developed  that  will  enable  unlike  investments  to  be 
compared  in  a  way  that  optimizes  a  company's  ability  to  maximize  its  strategic  investments  . 
Projects  can  then  be  compared  on  the  basis  of  their  scores  and  to  determine  the  optimal  value  for 
an  organization’s  IT  budget. 
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8.6.2.1  Benefit  Definitions 


Enhanced  ROI,  IRR,  orNPV 

The  commonly  used  ROI,  IRR,  or  NPV  calculations  may  require  special  consideration  when 
applied  to  IS  application  development  projects  because  they  typically  have  a  longer  useful  life 
than  non-IS  projects.  They  can  also  provide  benefits  that  can  be  leveraged  into  other  strategic 
investments  for  competitive  advantage,  and  can  improve  operating  efficiency  and  functional 
effectiveness  beyond  the  boundaries  of  a  single  firm  (e.g.,  EDI). 

Strategic  Match 

Strategic  matching  assesses  the  degree  to  which  the  proposed  project  corresponds  to 
established  corporate  and  business  unit  strategies  and  goals,  emphasizing  the  close 
relationship  between  IS  planning  and  corporate  planning.  Projects  that  form  an  integral  part 
of  the  corporate  strategy  will  be  assigned  a  higher  strategic  matching  score,  regardless  of  the 
economic  impact  calculation. 

Competitive  Advantage 

This  evaluates  the  degree  to  which  the  proposed  project  provides  an  advantage  in  the 
marketplace,  for  example,  inter-organizational  collaboration  through  EDI.  The  competitive 
advantage  dimension  requires  that  a  value  be  placed  on  a  projects  contribution  to  achieving 
one  or  more  of  the  following  goals:  altering  the  industry  structure,  improving  the 
organizations  position  in  its  existing  businesses,  or  creating  new  business  opportunities. 

Management  Information  Assessment 

Management  information  assessment  examines  a  project's  contribution  to  management’s  need 
for  information  on  core  activities,  e.g.,  activities  directly  involved  in  the  realization  of  the 
firm's  mission,  as  distinguished  from  support  and  accounting  activities. 

Competitive  Response 

Competitive  response  evaluates  the  degree  of  business  risk  associated  with  not  undertaking 
the  project,  which  includes  the  risk  of  losing  market  sham. 

Strategic  IM/IT  Architecture 

This  assesses  the  degree  to  which  the  proposed  project  fits  into  the  overall  IM/IT  direction  and 
assumes  the  existence  of  a  long-term  IM/IT  plan,  i.e.  an  architecture  or  blueprint  that  provides 
the  top-down  structure  into  which  future  data  and  systems  must  fit. 

8.6.2.2  Benefit  Measures 

Benefit  measures  gauge  the  accomplishment  of  a  result.  You  may  want  to  consider  benefits  in 
two  major  groups:  cost  reduction  and  value  enhancements.  Cost  reduction  benefits  result  from 
improved  operations  and  are  the  benefits  typically  identified  with  the  system.  Value 
enhancements  are  benefits  that  result  from  an  increase  in  services  to  the  organization  or  the 
organization's  clients  (e.g.,  timely  response  to  inquiries).  These  benefits  are  service 
improvements  not  provided  by  the  status  quo. 
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The  manager  or  analyst  can  directly  measure  many  benefits  in  monetary  terms.  For  example, 
projects  for  modernization  or  replacement  of  existing  equipment  can  generate  operating  and 
support  savings  relative  to  the  status  quo.  This  benefit  is  quantifiable  in  direct  monetary  terms. 

Replacing  a  particular  work  step,  function,  or  piece  of  equipment  is  another  common  benefit. 

For  example,  administrative  lead  time  or  delay  can  be  reduced,  resulting  in  fewer  resources 
needed.  A  remote  job  entry  station  can  replace  the  central  data  entry  operation,  with  a  resulting 
cost  reduction.  Productivity  and  accuracy  gains  through  on-line  entry  may  also  translate  into 
personnel  savings  (value  enhancement). 

Benefits  that  are  not  specifically  monetary,  but  quantifiable,  can  often  be  converted  into 
equivalent  monetary  values.  These  benefits  include  labor  savings  and  error  reduction.  An 
efficiency/productivity  increase,  typically  expressed  in  person-years,  is  a  benefit  whose  value 
includes  all  direct  and  indirect  labor  costs.  Direct  labor  costs  are  salaries  or  hourly  wages,  while 
indirect  labor  costs  include  allowances,  leave,  and  fringe  benefits  to  reflect  the  full  cost  of 
providing  a  person-year  of  labor.  Documented  personnel  reductions  are  the  best  evidence  of 
monetary  benefit. 

8.6.23  Risk  And  Uncertainty  Definitions 

Strategic  Uncertainty 

Strategic  uncertainty  is  an  assessment  of  the  degree  to  which  the  business  strategy  is  likely  to 
succeed. 

Organizational  Risk 

Organizational  risk  is  an  assessment  of  the  degree  to  which  an  IT  project  depends  on  new  or 
untested  non-IT  corporate  or  business  unit  skills,  management  capabilities,  or  experience. 

IM/IT  Infrastructure  Risk 

The  assessment  of  IM/IT  infrastructure  risk  is  essentially  an  environmental  assessment, 
involving  factors  such  as  data  administration,  communications,  and  distributed  systems.  It 
assesses  the  degree  to  which  the  entire  IM/IT  organization  is  both  required  and  prepared  to 
support  the  project. 

Definition  Uncertainty 

Generally,  this  assesses  the  specificity  of  the  business/user  objectives  that  are  communicated 
to  the  IT  project  personnel. 

Technology  Uncertainty 

Technology  uncertainty  assesses  a  project's  dependence  on  new  or  untried  technologies  which 
may  involve  a  single  technology  or  a  combination  of  new  technical  skillsets,  hardware,  or 
software  tools.  A  project  may  be  inherently  risky  if  it  requires  the  introduction  of  an  untried 
technology. 
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8.6.3  Strengths: 

IE  includes  non-monetary  aspects  of  benefit  which  are  usually  ignored  by  traditional  methods  of 
evaluating  IT  projects.  By  defining  value  and  risk  more  completely,  IE  helps  to  create  a  formal 
decision-making  process  around  IT  investments.  IE  assesses  both  business  feasibility  and 
technical  viability. 

8.6.4  Weaknesses: 

A  single  IE  evaluation  can  involve  many  complex  assessments  (e.g.  calculating  ROI). 

Combining  multiple  assessments  (e.g.,  business  strategy  and  technical  strategy)  can  be  even  more 
challenging.  It  is  not  always  clear  how  to  determine  the  relative  weights  for  the  various 
approaches. 

8.6.5  Examples 

Example  1.  Project  Evaluation  Using  Information  Economics 

1.  Determine  the  relative  weight  of  each  benefit  and  risk  category. 

2.  Score  each  projects  benefits  according  to  the  criteria  established  in  each  benefit  and  risk 
category.  (A  worksheet  is  needed  for  each  category.) 

3.  Compute  the  weighted  project  score  for  each  project. 


Benefit  Category: 

Weight 

Score 

Weighted  Score 

Economic  Impact 

10 

4.5 

45 

Strategic  Match 

2 

5 

10 

Competitive  Advantage 

2 

2 

4 

Management  Information 

2 

3 

6 

Competitive  Response 

1 

4 

4 

Strategic  IM/IT  Architecture 

3 

4 

12 

Cost  Category: 

Definition  uncertainty 

-2 

3 

-6 

Technological  uncertainty 

-2 

1 

-2 

IT  infrastructure  risk 

-2 

0 

0 

Final  Project  Score 

73 

Example  2.  Ranking  Information  System  Projects  Using  Information  Economics 

Projects  are  compared  by  weighted  scores  and  cost.  The  maximum  score  for  the  budget  available 
is  then  computed.  In  this  example,  even  though  Project  I  has  a  higher  total  score,  Projects  2  and 
3  together  have  a  higher  weighted  score  and  so  are  ranked  higher  given  a  budget  limitation  of 
$20M. 


Project  Number 

Weighted  Score 

Project  Cost 

Project  Ranking 

1 

87 

$20,000,000 

3 

2 

58 

$10,000,000 

1 
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8.7  Activity-Based  Costing 

Activity-Based  Costing  (ABC)  is  a  method  of  measuring  the  cost  and  performance  of  activities, 
products,  and  customers.  ABC  apportions  costs  to  products  or  services  according  to  the  actual 
activities  and  resources  consumed,  e.g.,  in  production,  marketing,  sales,  delivery,  and  service. 
ABC  was  developed  to  replace  conventional  cost  systems  since  automation  has  made  indirect 
costs  (a.k.a.  overhead)  a  significant  cost  factor  in  addition  to  labor  and  materials. 

8.7.1  Uses: 

ABC  can  be  used  to  accurately  measure  the  costs  of  both  activities  (e.g.,  systems)  and  cost 
objects,  (e.g.,  products  or  customers).  ABC  can  also  be  used  to  measure  the  performance  of  an 
activity.  As  a  result  of  this  measurement,  organizations  can  use  the  information  collected  to 
improve  performance  and  the  value  received  by  customers. 

8.7.2  Measurement  Approach: 

ABC  focuses  on  activities  that  use  an  organization's  resources.  An  activity  is  a  unit  of  work  in  an 
organization  that  consumes  resources.  Examples  of  activities  include:  providing  support  to 
customers,  maintaining  systems,  processing  orders.  Examples  of  resources  are:  salaries  of  staff 
performing  the  activity  and  the  people  supporting  it,  office  space,  materials,  direct  system  costs, 
system  overheads  etc. 

ABC  has  two  views:  a  cost  assignment  view  that  looks  at  the  costs  of  a  particular  activity;  and  a 
process  view  that  looks  at  the  performance  of  that  activity. 

The  Cost  Assignment  View  determines  the  costs  of  an  activity  and  relates  them  to  a  business' 
cost  objects.  A  cost  object  is  the  reason  a  company  Performs  the  activity  (e.g.,  a  product  or 
customer). 

Resource  costs  are  assigned  proportionately  to  the  activities  using  them.  Measures  are  identified 
for  the  frequency  of  use  of  an  activity,  e.g.,  for  the  "Hands  on  Support"  activity,  such  a  measure 
might  be  "number  of  trouble  calls". 

Each  activity  cost  is  prorated  using  a  measure  of  the  frequency  it  is  used  by  a  particular  cost 
object. 

The  Process  View  looks  at  the  same  activity  from  a  different  perspective.  Its  goal  is  to  measure 
the  performance  of  an  activity  and  to  identify  ways  performance  can  be  improved: 

Cost  drivers  are  identified.  These  are  factors  that  determine  the  workload  and  effort  of  an 
activity  and  the  resources  it  needs.  For  example,  the  credit  checking  activity  in  a  credit 
department  is  driven  by  the  number  of  sales  prequalifications.  An  activity  may  have  multiple 
cost  drivers. 
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Performance  measures  for  the  activity  are  defined.  These  indicate  how  well  an  activity  meets  the 
needs  of  its  internal  or  external  customers,  e.g.,  number  of  customer  complaints,  number  of 
errors. 

The  performance  measures  of  one  activity  often  become  the  cost  drivers  of  the  next  activity  in 
the  process. 

Performance  improvement  comes  from  using  both  views  and  has  three  steps: 

1.  Analyze  activities:  understanding  why  work  is  done,  and  how  well  it  is  done  will  help 
eliminate  waste  and  strengthen  position. 

•  identify  nonessential  activities 

•  analyze  significant  activities 

•  compare  to  best  practices 

•  examine  links  between  activities 

2.  Dig  for  drivers;  look  for  things  that  require  you  to  perform  nonessential  activities  or  to 
perform  below  par;  look  for  causes  of  waste 

3.  Measure  what  matters;  identify  the  mission;  communicate  the  objectives;  develop  the 
measures 

8.7.3  Strengths: 

ABC  is  a  process  of  continuous  improvement.  It  helps  set  strategic  priorities  and  implement 
chosen  strategies.  Companies  have  used  it  to  improve  profitability,  as  the  basis  for  directing 
investments  to  business  areas  with  the  greatest  improvement  potential,  and  to  manage  supplier 
relationships. 

8.7.4  Weaknesses: 

ABC  provides  only  an  "as-is"  snapshot  of  current  activities  and  conditions.  ABC  measures  costs 
during  the  current  period  only.  It  does  not  address  long-term  issues. 
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8.7.5  Examples 


Assigning  Costs  to  Activities  &  Cost  Objects 
System  Support  Overhead  Costs  ($1,000,000) 


Activity 

Proportion 

Cost 

Process  Changes 

25% 

$250,000 

Hands  on  Support 

50% 

$500,000 

Maintain  Documentation 

25% 

$250,000 

Cost  of  System  Support  per  System 

System  A  System  B 


Activity  Drivers 

Number 

Cost 

Number 

Cost 

Change  Requests 

75 

$187,500 

25 

$62,500 

Trouble  Calls 

500 

$294,110 

350 

$25,880 

Programs 

30 

$57,690 

100 

$192,300 

Cost  per  System 

$539,300 

$280,680 
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8.8  Integrated  Performance  Measurement 

Integrated  Performance  Measurement  (IPM)  focuses  on  the  strategic  measurement  of  a  system's 
success.  By  revealing  the  most  strategically  important  problems  to  be  solved  or  opportunities  to 
be  taken  advantage  of,  IPM  leads  to  continuous  improvement  a  long  strategic  lines. 

8.8.1  Uses: 

IPM  helps  users  to  develop  performance  measures  for  a  system  which  are  appropriate  for 
multiple  levels  of  an  organization  and  which  are  specifically  linked  with  organizational  strategy. 
IPM  can  help  IM/IT  managers  to  evaluate  their  current  IT  performance  to  learn  about  how  it 
could  be  improved. 

8.8.2  Measurement  Approach: 

Performance  measurement  of  IT  involves  evaluating: 

•  whether  the  system  is  effective 

•  whether  system  is  efficient 

•  how  the  enterprise  is  organized  to  do  the  work  surrounding  the  system. 

The  key  to  effective  system  performance  measurement  is  to  integrate  financial  and  nonfinancial 
measures.  This  can  be  done  by: 

•  using  accounting  measures  of  strategic  variables  when  looking  at  IT  performance  at 
the  top  levels  of  the  organization  (i.e.,  measures  that  track  results) 

•  using  project  measures  of  performance  at  the  lowest  levels  of  the  organization  (i.e., 
measures  that  track  actions) 

•  using  measures  that  track  the  relationship  between  actions  and  costs  at  middle  levels 
of  the  organization. 

To  improve  performance  measurement,  IPM  uses  a  two-part  Performance  Measurement 
Questionnaire: 

1.  Current  Performance  Assessment  identifies  the  current  importance  of  various 
business  performance  factors,  identifies  the  company's  current  ability  to  measure 
these  factors,  and  appropriate  IT  measures  can  then  be  designed  in  keeping  with 
desired  current  business  performance. 

2.  Future  Improvement  Assessment  identifies  the  long-run  importance  of  improvement 
in  various  business  performance  factors  and  identifies  the  support  by  current  systems 
and  other  company  investments  for  these  areas  of  improvement. 

The  effectiveness  of  current  performance  measures  in  measuring  performance  requirements  can 
then  be  determined.  Managers  may  also  choose  to  invest  in  IT  that  will  support  current  and 
future  performance  requirements. 
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8.8.3  Strengths: 

IPM  emphasizes  consistency  between  and  organization’s  strategies,  its  actions,  and  its 
performance  measures.  This  approach  communicates  the  organization’s  strategic  objectives  to 
every  part  of  the  company  and  lets  every  person  understand  how  his  or  her  actions  and 
investment  decisions  contribute  to  achieving  them.  IPM  addresses  the  future  needs  of  the 
company  by  continually  asking  questions  such  as,  "What  will  focusing  on  these  measures  lead  to 
today?  tomorrow?"  and  "What  next?" 

8.8.4  Weaknesses: 

The  usefulness  of  any  IPM  measure  depends  on  the  level  of  the  organization  and  the  time  frame 
looked  at.  Thus,  useful  measurement  systems  will  not  be  identical  for  any  two  locations,  units, 
levels,  or  event  time  periods.  This  makes  comparison  of  results  difficult. 

8.8.5  Examples 

Example  1.  Current  Performance  Assessment 

Managers  are  first  asked  to  rank  business  performance  factors  in  order  of  their  importance.  In 
this  study,  45  performance  factors  were  identified  and  ranked. 

Managers  are  then  asked  to  indicate  the  importance  they  place  on  measuring  various  performance 
factors. 

This  example  shows  seven  top  performance  factors  (marked  with  an  *),  which  are  not  being 
given  appropriate  importance  by  current  company  measures.  It  suggests  that  new  measures  need 
to  be  developed  for  these  areas  or  that  these  factors  need  more  emphasis. 


Current  Performance  Factor  Importance 

Current  Performance  Factor  Measurement 
Emphasis 

Top  25%  (high  to  low ,  ranked  1  to  11) 

Top  25%  (high  to  low,  ranked  1  to  11) 

1.  New  Product  Introduction 

1.  Safety 

2.  Safety 

2.  Back  Orders 

3.  Conformance  to  Specs 

3.  Yields 

4.  On-time  Delivery* 

4.  Direct  Labor  Productivity 

5.  Vendor  Quality* 

5.  Conformance  to  Specs 

6.  Sales  Forecast  Accuracy* 

6.  Department  Budget  Control 

7.  Research  Effectiveness* 

7.  Unit  Labor  Costs 

8.  Meeting  Production  Schedule 

8.  Variances 

9.  Environmental  Monitoring* 

9.  Meeting  Production  Schedules 

10.  Minimize  Environmental  Waste* 

10.  Controlling  Capital  Spending 

11.  Yields 

11.  Operations  Product  Costs 
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Example  2.  Future  Improvement  Assessment 


Managers  were  also  asked  to  evaluate  the  support  for  areas  of  improvement  that  was  provided  by 
IT  and  other  company  investments. 

31  business  areas  requiring  improvement  were  identified  and  ranked  in  order  of  importance. 

Ideally,  the  most  important  business  areas  requiring  improvement  would  be  those  areas  most 
supported  by  IT  and  other  investments.  In  this  case,  four  of  the  areas  most  important  to  the 
business  (marked  with  an  *)  were  among  those  receiving  the  least  support  from  IT  and  other 
investments.  This  indicates  that  the  agency  should  rethink  the  support  and  systems  it  provides  to 
the  top-ranked  business  areas. 


Improvement  Area 
Importance 

Improvement  Area  Support 

Top  25% 

Top  25% 

Bottom  25% 

(high  to  low,  ranked  1  to  8) 

(high  to  low,  ranked  1  to  8) 

(high  to  low,  ranked  24  to  31) 

1.  New  Product  Introduction 

1.  Customer  Satisfaction 

24.  Managing  Diversity 

2.  Sales  Forecasting 

2.  Environmental  Control 

25.  Job  Responsibilities 

3.  Competitive  Analysis 

3.  Direct  Cost  Reduction 

26.  Information  System 

4.  Responses  to  Changes  in 
Customer  Demand 

4.  Regulatory  Compliance 

27.  Response  to  Changes  in 
Customer  Demand* 

5.  Information  Systems 

5.  Labor  Efficiency 

28.  Performance  Measurement 

6.  Process  Technology 

6.  Inventory  Management 

29.  Competitive  Analysis* 

7.  Education  and  Training 

7.  Quality 

30.  Volume  Flexibility 

8.  Corporate  Information 
Management 

8.  Process  Technology* 

31.  Sales  Forecasting* 
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8.9  Information  System  Success  Categories 

These  categories  represent  a  comprehensive  overview  of  possible  types  of  Information  System 
success.  Within  each  category,  a  wide  selection  of  success  measures  are  given. 

8.9.1  Uses: 

These  categories  can  be  used  to  plan  a  complete  assessment  (i.e.,  both  financial  and  nonfinancial) 
of  an  individual  system's  impact  on  the  organization.  Individual  categories  can  be  used  to 
develop  a  thorough  assessment  of  a  system's  impact  in  a  particular  area  (e.g.,  impact  on  quality). 

8.9.2  Measurement  Approach: 

There  are  five  major  categories  of  information  system  success: 

1.  System  Quality  -  This  reflects  the  engineering-oriented  performance  characteristics  of  the 
system  e.g.,  reliability,  response  time. 

2.  Information  Quality  -  reflects  the  importance  and  utility  of  information  presented  by  the 
system,  e.g.,  information  accuracy,  relevance.  Use  -  Where  use  of  a  system  is  optional, 
the  amount  of  use  can  indicate  its  success  e.g.,  frequency  of  use,  number  of  requests  for 
particular  types  of  information 

3.  User  Satisfaction  -  This  measures  whether  users  actually  like  a  system  and  includes 
measures  of  both  general  satisfaction  (e.g.,  happiness  with  a  system)  and  specific 
satisfaction  (e.g.,  communication  with  IM/TT).  It  is  one  of  the  most  commonly-used 
measures  of  IM/TT  success. 

4.  Individual  Impact  -  This  addresses  how  the  system  has  improved  individual  performance 
e.g.,  decision-making,  productivity,  or  work  flow. 

5.  Organization  Impact  -  This  measure  includes  not  only  financial  "bottom-line"  impacts, 
but  also  impacts  on  productivity,  competitive  advantage  and  organization  structure. 

While  these  measures  can  be  looked  at  individually,  it  is  clear  that  they  are  interrelated  as  well. 
For  example,  systems  must  be  used  before  they  can  affect  individual  performance.  Individual 
performance  must  be  affected  before  organizational  performance  will  be  changed.  In  short  a 
thorough  assessment  of  a  system's  success  would  include  measures  from  each  of  these  categories. 

8.9.3  Strengths: 

These  categories  provide  a  comprehensive  view  of  the  measurement  of  IM/IT  Success. 

These  categories  help  organize  a  variety  of  measures  of  IM/IT  success  into  a  comprehensive 
whole  and  clarify  how  a  confusing  host  of  success  measures  fit  together. 

8.9.4  Weaknesses: 

Many  of  the  measures  presented  within  the  categories  have  not  been  studied  empirically. 

Different  studies  using  different  success  measures  selected  from  these  categories  will  come  up 
with  different  results,  making  it  difficult  to  compare  the  success  of  different  systems. 
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8.10  Value  Management  Framework 

The  Value  Management  Framework  is  a  method  of  evaluating  the  impact  of  a  system  from  a 
technical,  economic,  and  an  organizational  perspective.  The  framework  organizes  findings  at 
three  different  levels:  individual,  business  unit,  and  corporate. 

8.10.1  Uses 

To  assess  the  impact  and  benefits  of  a  system  after  implementation. 

8.10.2  Measurement  Approach: 

A  system  can  be  assessed  in  terms  of  its  impact  in  two  dimensions: 

•  Where  its  value  comes  from,  i.e.,  does  it  have  economic  benefits?;  did  it  change 
processes?;  did  the  technology  enable  additional  benefits? 

•  Where  the  value  occurred,  i.e.,  was  the  impact  at  an  individual  level,  a  business  unit  level, 
or  an  organizational  level? 

By  joining  these  two  dimensions  into  a  framework,  localized  measures  of  performance  can  be 
linked  with  organizational  performance.  For  example,  giving  a  sales  force  laptop  computers 
improves  tracking  of  sales  calls  and  expenditures  by  agent,  leading  to  better  management  of  the 
sales  process  by  region,  and  finally,  to  a  better  understanding  at  the  corporate  level  of  how  these 
measures  relate  to  improved  profit  margin. 

8.10.3  Strengths: 

This  framework  emphasizes  the  importance  of  assessing  a  system's  impact  at  multiple  levels  and 
from  several  perspectives.  It  points  out  that  financial  results  and  other  high-level  indicators  of 
success  are  the  end  products  of  many  intermediate  business  impacts.  Failure  to  understand  these 
provides  executives  with  no  ability  to  leverage  unanticipated  benefits  or  to  learn  from 
experience.  This  framework  is  a  structure  to  evaluate  the  intermediate  impacts  of  a  system. 

8.10.4  Weaknesses. 

The  framework  does  not  help  with  the  quantification  of  benefits. 
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8.10.5  Examples 

Evaluation  Of  A  Human  Resource  System 


Impact  Focus 

Source  of 
Value 

Technology  Enabling 

Impact 

Organizational  Process 
Outcome 

Economic  Performance 

Individual 

•  On-line  entry  of  job 
applications 

•  Improved  access  to  data 

•  Elimination  of  data 
entry  step 

•  Reduced  time  to  serve 
employee  request 

•  More  time  for  other  HR 
initiatives 

Business 

Unit 

•  Easier  access  to 
information 

•  Improved  accuracy 

•  Elimination  of  claims 
verification  process 

•  Improved  decision 
making 

•  Quicker  compliance 
with  government 
regulations 

•  Reduced  cost  of  placing 
employees 

Organization 

•  Elimination  of  14 
payroll  systems 

•  Data  and  process 
standards 

•  Ability  to  restructure 

•  Introduction  of  new 
benefits  programs 

•  Reduced  cost  of 
employee  benefits 
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8.11  Earned  Value 


8.11.1  Description 

The  earned  value  approach  is  dictated  in  DoD  Regulation  5000. 2-R  as  the  basis  for  the 
Cost/Schedule  Control  System  Criteria  (C/SCSC).  It  is  a  management  technique  that  relates 
resources  planning  to  schedules  and  to  technical  performance  requirements.  EV  requires  the 
contractor  to  plan,  budget,  and  schedule  all  authorized  effort  in  time-phased  “planned  value” 
increments  constituting  a  “performance  measurement  baseline”.  As  work  is  accomplished,  it  is 
“earned”  on  the  same  basis  it  was  planned. 

8.11.2  Uses 

To  assess  progress  toward  accomplishing  project  performance  goals  within  cost  and  schedule 
constraints. 

8.11.3  Measurement  Approach: 

The  earned  value  approach  measures  cost  and  schedule  variances  by  comparing  the  actual  costs 
(in  money  and  time)  of  work  performed  against  the  values  projected  in  the  project  baseline.  Work 
is  planned,  budgeted  and  scheduled  in  time-phased  increments  that  provides  an  immediate, 
objective  basis  for  comparing  actual  performance  with  projected.  Measuring  work  in  terms  of 
performance  capability  achieved  instead  of  effort  expended  can  provide  a  totally  integrated 
performance  measurement  system. 

8.11.4  Strengths: 

If  done  correctly,  the  earned  value  approach  provides  a  continuous,  integrated,  objective 
yardstick  of  all  three  factors  of  project  performance  measurement. 

8.11.5  Weaknesses. 

•  Relating  work  effort  to  capabilities  achieved  is  often  difficult. 

•  This  approach  is  best  suited  to  measuring  technical  performance  rather  than  mission 
capabilities. 


8-27 


8.12  The  Balanced  Scorecard 


8.12.1  Description: 

The  Balanced  Scorecard  is  a  set  of  financial  and  operational  measures  that  provide  a  balanced 
presentation  of  both  the  financial  and  operational  impacts  of  a  system  giving  senior  managers  a 
comprehensive  view  of  a  system's  value. 

8.12.2  Uses: 

To  assess  the  value  and  impacts  of  an  IT  investment. 

To  help  establish  performance  measures  that  relate  clearly  to  the  overall  objectives  of  a  system. 

8.12.3  Measurement  Approach: 

Senior  managers  need  to  know  the  impact  a  system  has  had  on  a  business'  operations  as  well  as 
its  financial  benefits.  Both  types  of  measures  are  equally  important,  and  no  single  measure  can 
address  both  these  dimensions.  The  Balanced  Scorecard  focuses  attention  on  four  areas  that  are 
most  critical  to  any  business.  Managers  using  it  to  evaluate  an  IT  investment  should  ask  the 
following  questions: 

1.  How  does  this  system  make  us  look  to  our  customers?  Managers  should  articulate  the 
system's  goals  in  relation  to  customers'  concerns,  i.e.,  time,  quality,  service  and  performance, 
and  price.  They  should  then  develop  measures  to  assess  the  systems  impact  accordingly. 

2.  How  does  this  system  affect  the  critical  operations  that  enable  the  company  to  meet  its 
customers'  needs?  These  would  include  processes  that  affect  cycle  time,  quality,  employee 
skills,  and  productivity. 

3.  How  does  this  system  enable  the  company  to  learn  and  grow?  Manners  should  assess 
how  the  system  improves  the  company's  ability  to  launch  new  products,  create  more  value  for 
customers  or  improve  operating  efficiencies. 

4.  How  does  this  system  contribute  to  an  Improvement  of  the  bottom  line?  Operational 
improvements  do  not  always  lead,  to  improved  financial  performance,  such  as  improved 
market  share  or  cash  flow.  However,  failure  to  do  so  should  warn  executives  that  they  must 
rethink  the  company's  systems  strategy  or  implementation  plans. 

The  Balanced  Scorecard  requires  managers  to  articulate  an  overall  vision  for  a  system  and  to 
develop  measures  that  are  designed  to  evaluate  how  well  this  vision  is  achieved.  As  a  result, 
measures  will  be  different  for  each  system.  Example  1  lists  some  sample  measures  of  ways  in 
which  a  system  might  have  both  operational  and  financial  impacts. 
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8.12.4  Strengths: 

The  Balanced  Scorecard  brings  together,  in  a  single  management  report  many  disparate  elements 
of  a  company's  competitive  agenda:  becoming  customer-oriented;  shortening  response  time; 
improving  quality;  emphasizing  teamwork;  reducing  new  product  launch  times;  and  managing 
for  the  long  term. 

This  approach  also  helps  managers  to  see  whether  improvement  in  one  area  has  been  achieved  at 
the  expense  of  another.  By  forcing  senior  managers  to  consider  all  important  measures  together, 
it  therefore  guards  against  suboptimization. 

The  measures  emphasize  strategy  and  vision,  not  control.  They  therefore  keep  executives 
looking  forward,  not  backward. 

8.12.5  Weaknesses: 

It  may  not  be  easy  to  develop  the  necessary  measures  of  performance  or  to  collect  the  appropriate 
data.  Systems  may  have  to  be  modified  to  capture  the  information  or  new  processes  put  in  place 
to  collect  it. 
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APPENDIX  A 

IT  Investment  Baseline/Performance  Agreement 

Purpose/Concept: 

DoD  must  establish  a  common  framework/understanding  for  baselining  IT  investments. 
The  Investment  Baseline/Performance  Agreement  is  the  recommended  approach.  The 
Investment  Baseline/Performance  Agreements’  performance  parameters  represent  the  minimum 
number  needed  to  characterize  the  major  drivers  of  operational  effectiveness,  suitability, 
schedule,  technical  progress  and  cost.  This  minimum  number  include  the  key  outcome 
measures  described  in  the  requirements  definition  document. 

The  Investment  Baseline/Performance  Agreement  will  be  used  to  enhance  stability, 
control  the  cost  growth  and  assess  how  well  IT  supports  the  achievement  of  mission  goals. 

This  framework  is  intended  to  fill  the  gaps  and  meet  the  requirements  of  ITMRA  and  OMB 
Circular  A-ll. 

This  framework  provides  the  CIO,  CFO,  functional  managers  and  PM/user  with  the 
means  to  obtain  timely  information,  assess  and  manage  risks,  and  maximize  the  value  of  IT 
investments.  The  intent  of  this  framework  is  to  track  an  IT  investment  from  cradle  to  grave 
(From  Mission  Need  through  disposal.).  The  IT  Investment  Baseline  will  be  used  during  a 
follow-on  IT  Investment  Operational  Assessment  Program  (To  Be  Established)  that  assesses 
performance,  quality,  compatibility,  interoperability  and  identifies  deficiencies. 

The  Investment  Baseline/Performance  Agreement  is  a  combination  of:  (1)  the 
Acquisition  Program  Baseline  (APB  required  by  DoD  5000-2R),  (2)  OMB’s  Circular  A-ll 
revision  3’s  baseline  requirements  (which  will  be  submitted  with  the  budget  for  programs),  and 
(3)  additional  measures  identified  in  ITMRA. 

The  Investment  Baseline/Performance  Agreement  will  be  used  to  track,  (1)  the 
accomplishment  of  the  Operational  Requirement  through  the  use  of  an  established  Program 
baseline  cost,  schedule  and  operational  performance  goals/outcomes  and  (2)  the 
accomplishment  of  the  acquisitions/contracts  through  the  use  of  an  established 
acquisition/contract  baseline  cost,  schedule  and  acquisition/contract  performance 
goals/outcomes  using  the  Earned  Value  Management  Framework.  The  assessment  of  the 
acquisition/contract  baseline  will  be  used  to  determine  the  successful  accomplishment  of  the 
Operational  Requirement/Program  baseline. 

The  Investment  Baseline/Performance  Agreement  is  recommended  as  an  update  or  appendix  to 
the  APB  (DoD  5000-2R  Acquisition  Program  Baseline). 
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INVESTMENT  BASELINE/PERFORMANCE  AGREEMENT 


OPERATIONAL  CAPABILITY 


Mission  Goal(s):  Program  Name:  Program 

Objectives: 

(Functional  Area  Supported) 


Current  Operational  Capability: 


Enhanced  Operational:  Enhanced  Operational  Capability 

Capability  Requirements  Performance  Measures  of  Effectiveness 

&  Efficiency:  Define  Measures 

Benefits  Assessed: 

•  Return  on  Investment: 
(quantitative/qualitative) 

•  Net  Present  Value: 

•  Quality 

•  Internal  Rate  of  Return: 

•  Compatibility 

•  Interoperability 
Annual  Maintenance: 

•  Annual  Operations  &  Support  Costs 
Risks  &  Deficiencies: 

•  Failed  Performance  Goals/Measures 


Strategic  Match:  {Strategic  matching  assesses  the  degree  to  which  the  proposed  project 
corresponds  to  established  corporate  and  business  strategies  and  goals,  emphasizing  the  close 
relationship  between  FT  planning  and  corporate  planning} 

Conduct  Operational  Benefits  Assessment  Program/  Post  Deployment 
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PROGRAM  COST  &  SCHEDULE 


Program  Costs: 

Then  Year 
Base  Year 

Approved  Budget  Amount 

Average  Unit  Procurement  Cost 

Life  Cycle  Cost 

Sunk  Cost 

Cost  to  Complete 


Program  Schedule: 

Program  Initiation 

Major  Milestone  Decision  Points 

Initial  Operating  Capability 

Critical  System  Events 
FY  19PY  Accomplishments 
FY  19CY  Planned  Program 
FY  19BY1  Planned  Program 
FY  19BY2  Planned  Program 


Risks  &  Benefits: 


Risks  Assessed: 

•  Strategic  Uncertainty  {assessment  of  the  degree  to  which  the  business  strategy  is  likely  to 
succeed} 

•  Technological  Uncertainty  {assesses  a  project’s  dependence  on  new  or  untried  technologies 
which  may  involve  a  single  technology  or  a  combination  of  new  technical  skillsets,  hardware 
or  software  tools.  A  project  may  be  inherently  risky  if  it  requires  the  introduction  of  an 
untried  technology} 

•  IT  Infrastructure  Risks  {essentially  an  environmental  assessment,  involving  factors  such  as 
data  administration,  communications  and  distributed  systems.  It  assess  the  degree  to  which 
the  entire  IT  organization  is  both  required  and  prepared  to  support  the  project} 

•  Organizational  Risks  {an  assessment  of  the  degree  to  which  an  IT  project  depends  on  new 
or  untested  non-IT  corporate  or  business  skills,  management  capabilities  or  experience} 

Strategic  Impacts  : 

•  Management  Information  Assessment  {examines  a  project’s  contribution  to  management’s 
need  for  information  on  core  activities,  e.g.,  activities  directly  involved  in  the  realization  of 
mission  goals} 

•  Competitive  Response  {evaluates  the  degree  of  business  risk  associated  with  NOT 
undertaking  the  project} 

•  Strategic  Information  Systems  Architecture  {assesses  the  degree  to  which  the  proposed 
project  fits  into  the  overall  Information  Resource  Management  Strategic  Planning  direction, 
and  assumes  the  existence  of  a  long-term  IT  plan} 


Variance  From  Program  Baseline  Goals 
Variance  in  total  program  cost  -  (10%  or  more  above  baseline) 
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Variance  in  total  program  schedule  -  (10%  or  more  behind  baseline) 

Variance  in  operational  capability  performance  indicators  -  (report  on  whether  the  program 
has  deviated  at  all  from  performance  goals  and  measures  specified  in  this  baseline/contract) 

Corrective  Actions: 

Describe  proposed  corrective  actions  for  each  variance  form  baseline  goals  noted  above. 

Proposed  Revisions  to  Baseline  Goals: 

Describe  and  justify  any  revisions  to  goals  set  forth  in  the  original  baseline  agreement. 
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ACQUISITION/CONTRACT  BASELINE 


Total  Procurements: 

Earned  Value  Management  Framework  {this  framework  is  required  for  every  Primary 
Acquisition,  all  sub-acquisitions  will  report  the  progress  of  their  acquisition  baselines  to  the 
Prime.  Earned  Value  needs  to  be  tracked  over  time  to  be  useful  to  decision  makers  (i.e.  each 
contractual  vehicle  needs  to  be  evaluated  monthly} 

Acquisition  Baseline  Cost  &  Schedule  Goals  {show  the  dollar  amount  of  the  project  that  will 
be  completed  each  year.  Identify  and  discuss  how  many  months  it  will  take  to  complete  the 
acquisition,  important  components,  and  important  milestones  within  that  time} 

FY  19PY  Accomplishments 
FY  19CY  Planned  Program 
FY  19BY1  Planned  Program 
FY  19BY2  Planned  Program 
Cost  Performance  Index  (CPI) 

Schedule  Performance  Index  (SPI) 

Acquisition  Baseline  Performance  Goals 

{ summarize  the  performance  goals  of  the  acquisition  as  stated  in  the  statement  of  work  and 
describe  the  relationship  of  the  acquisition  to  the  overall  operational  capability} 

Variance  from  Acquisition  Baseline  Goals 

1.  Variance  in  Acquisition  Cost:  {identify  whether  the  current  acquisition  cost  estimate  is 
10  percent  or  more  above  the  baseline  goals.  Discuss  and  give  reasons  for  the  variance} 

2.  Variance  in  Acquisition  Schedule:  {identify  whether  the  current  schedule  estimate  is  10 
percent  or  more  behind.  Discuss  and  give  reasons  for  the  variance } 

3.  Variance  in  Acquisition  Performance  goals:  {identify  whether  performance  goals  deviate 
at  all  from  the  performance  goals  stated  in  the  statement  of  work.  Discuss  and  give 
reasons  for  the  variance } 

Proposed  Revisions  To  Acquisition  Baseline  Goals 

{Program  Manager  may  propose  revisions  to  the  acquisition  baseline  cost,  schedule  and 
performance  goals  if  current  estimates  indicate  they  are  not  achievable.  The  proposed  revisions 
must  be  justified,  with  an  estimated  probability  of  achieving  the  new  goals.  The  CIO,  CFO  and 
Functional  Sponsor  must  approve  any  changes  to  the  acquisition  baseline  goals. } 

Effectiveness  and  Suitability  Performance  Indicators: 

Technical  Evaluation  Criteria 

Technical  performance  measures  to  be  evaluated  during  the  test  and  evaluation  phase  of  the 
system  development;  also  documented  in  the  Test  and  Evaluation  Master  Plan. 
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VALIDATION 


CIO_ 

CFO 


Functional  Proponent 


Project  Manager 
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DEFINITIONS 


Program  Costs  Definitions: 

Then  Year:  Dollars  that  include  the  effects  of  inflation  or  escalation  and/or  reflect  the  price 
levels  expected  to  prevail  during  the  year  at  issue. 

Base  Year:  A  reference  period  which  determines  a  fixed  price  level  for  comparison  in  economic 
escalation  calculations  and  cost  estimates.  The  price  level  for  the  base  year  is  1.00. 


The  following  is  an  example  of  both  Base  Year  (BY)  and  Then  Year  (TY): 

TY  BY  TY  TY 

FY95 _ FY96 _ FY97 _ FY98 

Escalation  Index:  .985  1.00  1.15  1.30 

Fiscal  Impact:  $98.5  $100.0  $115.0  $130.0 


Approved  Budget  Amount:  The  total  approved  budget  authority  provided  by  law  to  enter  into 
obligations  that  will  result  in  immediate  or  future  outlays.  The  budget  amount  is  provided  by 
fiscal  year  and  appropriation. 

Average  Unit  Procurement  Cost:  Includes  recurring  flyaway,  rollaway,  sailaway  costs  (including 
nonrecurring  production  costs)  adjusted  for  data,  training,  support  equipment,  and  initial  spare 
costs. 

Total  Procurement  Quantities:  The  total  number  of  fully  configured  end  items  a  DoD  component 
intends  to  buy  through  the  life  of  the  program.  This  quantity  may  extend  beyond  the  FYDP  years 
but  shall  be  consistent  with  the  current  program. 

Life  Cycle  Cost:  The  total  cost  to  the  government  of  acquisition  and  ownership  of  that  system 
over  its  useful  life.  It  includes  the  cost  of  development,  acquisition,  support,  and,  where 
applicable,  disposal. 

Program  Schedule  Definitions: 

Program  Initiation:  The  start  of  a  defined  effort  funded  by  RDT&E  and/or  procurement 
appropriations  with  the  expressed  objective  of  providing  a  new  or  improved  capability  in 
response  to  a  stated  mission  need  or  deficiency. 

Major  Milestone  Decision  Points:  The  point  when  a  recommendation  is  made  and  approval 
sought  regarding  starting  or  continuing  (proceeding  to  the  next  phase)  an  acquisition  program. 
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Milestones  are:  0  (Concept  Direction),  I  (Concept  Approval),  II  (Development  Approval),  HI 
(Production  Approval),  and  IV  (Major  Upgrade  Decision). 

Initial  Operating  Capability:  The  first  attainment  of  the  minimum  capability  to  effectively 
employ  a  weapon,  item  of  equipment,  or  system  of  approved  specific  characteristics,  and  which 
is  manned  or  operated  by  an  adequately  trained,  equipped,  and  supported  military  unit  or  force. 

Critical  System  Events:  Interim  milestones  in  the  acquisition  process  that  review  a  systems 
capability,  either  operational,  technical,  or  other,  that  must  be  questioned  before  a  system’s 
overall  suitability  can  be  known,  and  which  are  of  primary  importance  to  the  Milestone  Decision 
Authority  in  reaching  a  conclusion  on  allowing  the  system  to  advance  to  the  next  phase. 

Other: 


Earned  Value  (EV):  A  management  technique  that  relates  resources  planning  to  schedules  and  to 
technical  performance  requirements.  EV  requires  the  contractor  to  plan,  budget,  and  schedule  all 
authorized  effort  in  time-phased  “planned  value”  increments  constituting  a  “performance 
measurement  baseline”.  As  work  is  accomplished,  it  is  “earned”  on  the  same  basis  it  was 
planned. 

Return  On  Investment  (ROI):  Net  income  divided  by  investment,  or  operating  revenues  divided 
by  operating  costs. 

Net  Present  Value:  A  time  value  of  money  tool  intended  to  support  the  choice  of  cash  flow 
alternatives.  The  present  value  of  a  series  of  future  cash  flows  is  the  value  that  it  would  be 
necessary  to  invest  at  the  present  time  at  a  specified  interest  rate  to  be  able  to  make  the  future 
cash  disbursements  and  receipts  of  the  cash  flow  and  completely  exhaust  the  investment  balance. 

Internal  Rate  of  Return:  The  discount  rate  that  results  in  a  present  value  of  zero  in  cash  flow. 
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